The analyst is the main stakeholder of the project. All team members: developers, testers, designers, and even managers go to analysts with questions at any stage of the project. The better the analytical part of the development is worked out, the calmer is the atmosphere in the team on the one hand, and the lower is the risk of rollback actions in meeting the customer’s expectations on the other hand.
Often, the analyst enters into the project earlier than everyone else (sometimes even at the pre-sale stage) and leaves from the project later than everyone else, in order to ensure stable functioning of the system on the customer’s premises. Only analysts are deeply immersed into the context and needs of the customer, and their unique role is to translate business requirements into development tasks.
Artificial intelligence (AI) is actively gaining momentum nowadays. In the custom software market, the prevailing demand for information systems (IS) with AI components becomes commonplace. In this situation, the question arises about the readiness of market participants to meet modern customer requirements.
This article offers to look at the main directions of change that await any development team in this regard. In particular, from the position of analysts as mandatory participants in any stage of the project. According to SDLC (Software development lifecycle), the system is created sequentially: requirement analysis, design, development, testing, deployment and support. What kind of work transformation should participants prepare for each stage?
The analytics stage traditionally involves active work with business requirements, their clarification, and discussion with the customer. Exclusively on this basis, the concept, design project and requirements for the architecture of the future system are consistently formed. Here, we define the high-level data model, the main logic of data processing and subsequent presentation in the UI.
But what to do with the data that will be generated, for example, by the LLM (Large Language Model)? How to integrate them into the general logic of the system? And most importantly — what kind of data will be there and what level of quality? All these questions should be answered already at the first stage of the project.
Thus, the boundaries of the analytical development stage are expanded, supplementing the package of business requirements with the capabilities and limitations of the LLM, i.e., creating a new category of system requirements.
As a rule, the AI team includes its “own” analysts (data analysts) or Data Scientists, who provide analytical development of scenarios for the future behavior of the model.
However, it is important for the project analyst to know in details the boundaries of the model’s operation in order to level out possible omissions of the LLM on the one hand, and, on the other hand, to exclude the automation of duplicate processes.
For example, the project includes the AI that checks the text to identify any entities. Let’s say, mentions of document names. In this case, there is no need to require “traditional” search features, the LLM will do it. However, it is necessary to know which methods will the model use to perform a task, and what is the declared quality level of the methods used. It will give to the analyst the information for modeling of data transfer from the LLM to post-processing. Thus, it is necessary to understand both the strengths and weaknesses of the AI component, ensuring system integrity when delivering information to users.
The identified AI requirements move to the system design stage. Just by the fact of their appearance, and accordingly, the need to elaborate on this set of requirements of a new kind, changes the team’s work at this stage too.
For the data model to be designed, the data source (or supplier) should be determined. Data is classified as external (coming from external IS or through input) and internal, which is created through calculations by the system itself. In other words, internal data is the result of the work of the created automated features. At the same time, the result of the work of the LLM should also be considered as internal system data, which are subjected to subsequent processing.
Thus, when designing system features, there are additional conditions and cases that arise and which are focused on the specifics of the model’s operation (with its pros and cons).
It is assumed that we approach the development stage with complete design documentation that contains answers to all questions from developers to start programming.
The other criterion for design quality can also be the immutability of the system architecture and logic throughout the entire project period, i.e., the absence of contradictions and the team’s desire to rework or otherwise change the project.
The programming stage is perhaps the most sensitive to possible logical conflicts. It is at this stage that inconsistencies in the interaction between system components potentially manifest themselves, especially on a “special” data set.
Therefore, the analyst must be prepared for this, and should know and understand well the logic of the LLM operation, what kind of surprises it can create.
As a rule, two main solutions are used:
1) If the data from the model can be formalized (for example, checked against a template), it is advisable to provide the data to subsequent processing in the description of the corresponding algorithms.
2) If it is assumed that the model produces data of various non-unified properties, then “troubleshooting” will move to the user level. To do this, it is necessary to provide the output with extremely detailed information in the interface, based on which the user can clearly understand whether he or she can trust the results of the system.
At the same time, let’s be realistic, it is impossible to predict the result of LLM work at 100%. That is why such a risk zone requires special attention, which can be expressed in the preparation of “special” data and taking into account the specifics of their processing in the description of the development tasks.
Gradually, we come to the testing stage. It is assumed that the described features of the AI system development will receive a corresponding continuation in test cases. The QA engineer, with the support of the analyst, will have to work out the “uncertain” scenarios for testing the system, i.e. scenarios that assess the stability of the system as a whole with unpredictable model behavior.
In the example above, if the searched document name is a sequence of characters that is unambiguously checked by the mask, then testing will be limited to enumerate possible combinations within the specified boundaries in order to assess the quality of the post-processing algorithms. You can also “hide” the acceptable value in a nested object of the text being checked (for example, in a table, headers, footers, etc.) and check the success of the search.
Otherwise, it is advisable to provide a chain of characters in the control text that the model can potentially accept as a document name. In this case, the evaluation of the completeness and availability of information in the UI should be done by whether it is clear to the user that the model mistakenly accepted our data as the target value.
By the testing stage, the team already understands the maximum quality level of the model, which will obviously be less than 100% (regardless of the project). In case of non-compliance with the planned quality level, we have already provided the algorithms that are focused on processing the unpredictable model data. In such a probabilistic coordinate system, it is necessary to plan activities to assess the quality of the developed AI system.
The final stage of the project is the deployment and training of users, which is carried out on the customer’s premises. Here, a significant range of new diverse issues arises that have not been previously discussed with customers. For example, the features of the infrastructure and its readiness to provide significant computing resources for the correct operation of the model.
Another example: the need for additional training of the model, the need for which will objectively arise after some period of operation of the system.
Of course, these and other related issues are not in the list of direct tasks of the analyst. However, it is advisable to include a procedure to follow when such a situation arises, in the operational documentation for the system.
An important aspect of the documentation is the description of the roles of the system users who perform the supporting functions. If previously such functions were assigned to the System Administrator (we mean all categories of administrators), now for AI systems it is recommended to additionally consider the role of the AI Engineer. In general, his responsibilities include additional training of the model in the corresponding part of the system.
Description of operations for working with prompts can be included both in the administrator’s guide (if the document is aimed to two categories of users) and in a separate document — the AI Engineer’s guide. Everything depends on the project and the customer’s plans for its development. However, it is advisable to work out and discuss with the customer all such issues at the start of the project, even though they are resolved at the final stage of the project.
This article does not claim to be a complete list of issues that artificial intelligence dictates to analysts. But we believe that we have covered the basic aspects and general principles of the necessary changes in work. After all, as noted at the beginning of the article, for the customer the analyst is an indicator of the team’s expertise in the field of automated procedures and a regulator of compliance with the expected results of the project.
