Stages of the data migration process in IP implementation projects. The Largest Migration Systems of the World: Common Trends and Differences Migration Information System

Any business strives to spend as little as possible, including on IT. One way to save money is to move information systems to the cloud. When transferring data to a virtualized environment, it is necessary to take into account the requirements of the legislation on the storage and processing of personal data. These requirements apply to any organization.

Vladimir Lebedev, director of business development at Stack Group, told us about the intricacies of the migration process.

Legal requirements

In 2006 was adopted 152-FZ "On Personal Data", which is designed to protect individuals in the automated processing of personal data. Last year, a package of amendments on the localization of personal data in Russia came into force, which, according to the authors, should increase the level of information security within the state and stimulate the Russian market of technological solutions and the information security market.

By law, businesses are required to collect, store and process personal data in the Russian Federation. All requirements are exactly the same for both Russian and foreign companies, if their activities are directed to the territory of Russia. At the same time, it is possible to transfer personal data outside the country, but they must be immutable, and their volume should not exceed the volume in Russian databases.

Who is the law for?

Each organization acts as an operator of personal data, such as its employees, customers, partners, so all companies in all industries fall under the law. However, potential inspections primarily threaten companies whose business activities are based on the processing of personal data: banks, insurance companies, companies providing medical services, and the like.

The inspection plan of Roskomnadzor for 2016 included: the largest software companies, international banks, network trading companies and online stores.

Difficulties in transferring personal data for international companies

Often, global international players, in order to comply with legal requirements, are forced to transfer to Russia not only the data itself, but also all related systems: it is extremely difficult for companies with centralized IT systems to change the architecture of an information system to one local legislation, because initially it did not have spaced information processing centers. Significant investments are required to rebuild information systems.

Virtualization

Moving to the cloud is less expensive than buying and installing hardware. At the end of 2014, prices for Russian clouds were on average 15–30% higher than for European ones, and at the end of 2015, on the contrary, our prices became 20–30% lower: the exchange rate and the relative cost of placement in Russian data changed. -centers.

Operators began to offer a more mature product, and foreign companies got the opportunity to choose among virtualization systems, price segments, and there were opportunities for connecting Russian and Western infrastructures.

For foreign companies, virtual infrastructure is a reliable and understandable solution. Many companies transfer information exclusively to a virtualized environment, which can also accommodate the necessary means of protecting personal data. When moving, foreign companies, of course, expect the operator to comply with the requirements of information protection regulators.

Storage Migration Risks

The migration processes themselves from one system to another carry risks. First of all, the established level of customer service may be violated. Communication may also be interrupted and the work of units whose activities are tied to automated systems may be suspended, conditions may arise when it is impossible to comply with information security requirements, and others. technical problems. Therefore, it is important that providers are prepared to ensure business continuity using reliable and time-tested solutions for the transfer of information systems.

Stages of cloud migration

General principles of service migration, i.e. transfer operating systems responsible for the operation of this service in a virtualized environment, consider the example of a solution VMware vSphere.

It is desirable to divide the architecture of information systems of a company into services that perform specific tasks. Usually, everything is moved to the cloud, except for services that are incompatible with cloud technology for technical reasons (RISC architecture) and those that cannot be moved due to a license.

Next, you need audit of information systems. At this stage, the composition of services is determined (which operating systems belong to a particular service), as well as their connectivity. The main difficulty lies in the variety of source operating systems and the physical architecture of the servers on which they run. Based on this information, a migration plan is drawn up taking into account current business processes: the requirements for the connectivity of physical and virtual infrastructures, the order of migration are determined, and the allowable “migration windows” are set. It is important to remember that you cannot upgrade versions of software products or operating systems during migration. Simultaneously with the migration, only revision of computing resources (CPU, RAM, HDD) is allowed.

As a rule, the utility is used for migration VMware converter, which works effectively when transferring OS family Microsoft Windows(but the migration of services running in these operating systems has its own nuances). But due to the peculiarities of Linux file systems, in about 40% of cases, after the end of the VMware converter, the virtual machine may not start. If LVM is used in Linux, then you need to deploy a new OS instance from the provider template in a virtual environment and then transfer data, software products and internal services.

For any type of OS, there are general conditions that make migration difficult: firstly, the data storage method that makes direct migration impossible is dynamic disks on Windows or LVM on Linux, and secondly, the difficulties due to the use of software and hardware RAID arrays. So, even an accurate data transfer alone does not guarantee that the virtual machine will start successfully. On a physical server, the operation of virtual machines is provided by a hypervisor - an OS that divides a physical server into several virtual machines that can work simultaneously and use the same physical resources. Naturally, the set of virtual hardware of the hypervisor does not match the hardware of the physical server on which the OS was running before the migration. Accordingly, due to the difference in drivers, there are many differences in access to this equipment.

Migration of ADDS and MS SQL without stopping services

Almost always, a business needs a number of services to remain available during a migration. At the same time, often migration without stopping the service recommended as the most reliable. Therefore, let's consider the features of non-stop migration of the most popular Microsoft OS services: Active Directory Domain Services (ADDS or AD) and Microsoft SQL (MS SQL). To migrate Active Directory without stopping the service, the following algorithm is used:

  • Network connectivity is established between the physical hardware and the virtualized environment. Typically, this is a site-to-site VPN - it creates a logical network on top of another network. At the same time, traffic can be protected by encryption using IPsec protocols.
  • In the cloud, we deploy new virtual machines from the template, where we configure AD domain controllers and add them to the forest.
  • We replicate the Active Directory database over the network via VPN from working controllers on the side of physical equipment to the cloud ones.
  • After data replication, we reassign the masters of the roles of operations to cloud controllers and remove the roles of domain controllers from the servers.
  • Then we check the operation of the services and disable Accounts old controllers and physical hardware.

Algorithm MS SQL migrations more complicated, since MS SQL is usually used in a multi-tier service as a backend. In DNS records in applications that use databases (in MS SQL clients), you must manually specify the new location of the database. Therefore, downtime cannot be completely eliminated, but it can be minimized. There are also non-stop MS SQL migration mechanisms, these include Mirroring And alwayson, but their use is not always justified. AlwaysOn is only available in expensive Enterprise-level editions, while Mirroring must be supported by MS SQL clients. In addition, to use Mirroring mechanisms, additional configuration of all MS SQL clients is required.
Consider the most common option for migrating MS SQL to the cloud:

  • Network connectivity between the cloud and physical equipment is configured.
  • We make sure that the MS SQL database recovery model is complete, then you can make and transfer a full backup, and then synchronize both databases, transferring copies of transaction logs.
  • Deploying in the cloud virtual machine from a template in which we install and configure a new MS SQL server.
  • We create a full backup of the MS SQL server database running on a physical server, then restore it in the cloud, while the method of transferring the backup depends on the file size and network bandwidth - we move it on a physical medium or copy it over the network.
  • After restoring the database in the cloud, we make a copy of the transaction logs and restore them in the cloud as well.
  • During the “migration window”, we stop the MS SQL server running on the physical hardware, create and restore the last minimal copy of the transaction logs in the cloud, start the MS SQL server in the cloud and switch clients to a new database location.
  • We check the work of services, turn off the physical equipment.

For each service and service, there are many ways to migrate, depending on the conditions. A service provider will help you avoid the difficulties that may arise during migration.

Information Security

Companies must not only transfer data to Russia, but also ensure an appropriate level of information security for their storage.

In the list of documents that define the requirements and describe the measures necessary to protect information systems, in the first place, of course, is the law itself, which establishes general requirements. Specific actions are regulated by decrees of the Government of the Russian Federation and normative documents FSTEC ( federal Service for technical and export control) and the FSB (Federal Security Service).

Requirements for the storage of personal data

The requirements for the technical protection of confidential information and the provision of information protection services are quite clearly formulated. The tools for their implementation are diverse. In particular, these can be firewalls, intrusion detection systems, security analysis tools, anti-virus protection, virtualization environment protection tools. The market offers a wide range of information security tools - both Russian and foreign vendors. There is already law enforcement practice, since the law has been in force since 2007. In general, the approach to regulation in Russia differs from, for example, the European approach. Thus, in Russia, failure to comply with prescribed information security requirements leads to liability. And in the West, a company can independently determine how to fulfill the requirements, and liability arises only if illegal actions are committed with personal data.

Infrastructure Requirements

Regulators also impose certain requirements on infrastructure. For example, there is certification of built automated systems that provide information protection. The certificate is issued by an independent laboratory, which confirms that the infrastructure is ready and meets all the requirements for the security of information systems. This certificate is optional for commercial organizations, but it makes it possible to understand that the data center has the necessary protection and is able to ensure the security of company data.

There are also international ISO standards that regulate the construction of an information security management system ( a set of ISO 2700x standards). Many foreign companies meet these standards.

In addition, an outsourcing model for data protection is applied. Companies providing such services must have the necessary licenses and competencies.

INFO

In China, a full copy of personal data must be stored within the country, and any banking data is generally prohibited from transferring outside of it.

Transfer forecast

It is rather difficult to calculate exactly how much data is to be transferred to Russia, but, based on the occupancy of the data center market, we can say that there is enough capacity to localize data in accordance with the law. For example, there is an overcapacity in the market of the Moscow region: the total capacity is about 27 thousand racks, and almost 40% of them are free. Many data centers have high availability areas. It should also be noted that the density of data in one rack may vary depending on the equipment. Today, a single unit of a server rack processes significantly more information than a few years ago.

At the same time, demand is uneven: a year ago there was a rather serious rise, now the number of applications from foreign companies has decreased. Some companies have already moved and meet the requirements, while others are waiting, for example, for the results of regulatory audits (they will affect the formation of the second wave).

Migration to Russian history. The problems of migration today cause a very ambiguous reaction in Russian society. Meanwhile, the territorial movement of the population is truly a worldwide phenomenon. Labor migration is an essential component of the global labor market. According to the UN, in 2015 there were 244 million migrants in the world, which amounted to approximately 3% of the world's population.

Meanwhile, the real scale of global migration flows may turn out to be much larger, since the above estimate does not take into account many illegal, or rather illegal or irregular migrants, both international and internal. At the same time, it should be noted that the currently popular term “illegal” is technically incorrect, since it mainly refers to people crossing the border or taking other steps in violation of existing legislation with the ultimate goal of getting a job, and not engaging in criminal activity as such. In addition, migrants often become "illegal" not of their own free will, but as a result of both targeted or ill-conceived actions of the government bodies of the receiving countries, and the use of corruption schemes by local employers and officials at the grassroots levels of government.

In general, in order to remain “in status”, a migrant must fulfill three basic conditions: legally cross the border of the host country, stay in it and obtain the necessary registration, and also get a job in accordance with the requirements of local legislation. Meanwhile, often migrants lose their legal status against their will as a result of discriminatory legislation, contradictions in local laws, or the actions of corrupt representatives of migration and law enforcement agencies. Suffice it to say that there are more than 229 million internal migrants in China alone, including about 200 million who changed their place of residence without obtaining appropriate permission from government agencies and, in fact, found themselves without a regular legal status in their own country.

And in this sense, Russia's migration problems are not too different from those faced by other major migrant-receiving countries. A distinctive feature of the Russian Federation has been the rapid change in recent decades of its role in the global migration chain, although in this respect there are interesting and very instructive parallels in world practice, primarily in a number of developing countries. Similar trends in the dynamics of migration flows and a plurality of functional roles are observed, for example, in other BRICS countries, which play an increasingly prominent role in the migration sphere. Suffice it to say that in the period 2000-2010. Russia ranked 3rd in the world in terms of annual immigration (389 thousand people), while South Africa was in 6th place (247 thousand). India and China occupied the 3rd and 4th places among the states with a negative migration balance with annual absolute losses of 490 thousand and 418 thousand people, respectively. while also acting as host countries. It is no secret that for centuries Russia has been distinguished by its closed borders and strict state control over migration, primarily external. At the same time, there was a very intensive movement of the population, especially ethnic Russians, from central regions to the ethnic periphery of the country, and the most important part of this migration flow was made up of professional elites. These processes were even more intensified during the Soviet period and proceeded under strict state control. As a result, by the time of the collapse of the USSR, a significant part of Soviet citizens lived outside their titular republics - out of the total population of the USSR, which exceeded 289 million people in 1989, more than 54 million only among the titular population of fifteen union republics.

In particular, more than 25 million Russians and 9 million representatives of the titular nations of the Russian regions lived outside of Russia [National composition…, 1991]. Bloody ethnic conflicts, discrimination in a number of newly formed states, the desire to return to their historical homeland in the current new conditions, as well as the fear of losing the citizenship of “their” countries led in the first post-Soviet years to the formation of significant flows of return migration, including to Russia. Thus, the situation in the post-Soviet space has changed radically, Russia has become the center of the second largest immigration system in the world after the United States: 12.3 million people in the country were born outside of Russia. The United States, which is the leader in this indicator, is home to 42.4 million people born in other countries. In the case of Russia, the vast majority of immigrants (that is, people born outside of Russia) came to it from other post-Soviet countries (“new abroad”). However, there are also migration flows from other states. Of particular importance in this regard is labor migration from China, although its real scale is much smaller than the sensational figures quoted by politicians and the press. In general, at this stage, we can talk about the complete dominance of Russia within the Eurasian migration system, although there are other visible flows, for example, labor migration to Kazakhstan. The relatively small flows of highly skilled migrants in the post-Soviet space are also important. At the same time, for the first time since the Civil War, there was also an impressive legal emigration: since 1991, about 1.3 million Russian citizens have received permission to leave the former USSR for permanent residence [Population of Russia, 2006]. This process was accompanied by a large-scale departure to the West of labor migrants, including highly skilled ones.

In addition, the lifting of the "Iron Curtain" led to the fact that a stream of people headed to Russia, seeking to get to the West. Thus, it began to play three fundamentally different roles in the global migration chain at once: a country that receives immigrants, a country of emigration (mainly to the countries of the “old abroad”), and a country of transit for those who are trying to get through its territory to the West. This phenomenon required quick creation the legislative framework for migration policy and the re-formation of the structures of the migration service, since in Soviet times external migration had a limited scale and was tightly controlled by the state. Not only emigration, but also the phenomena of immigration and the mass reception of refugees were rare phenomena. Article 38 of the USSR Constitution of 1977 declared that “the USSR provides asylum to foreigners persecuted for protecting the interests of workers and the cause of peace, for participating in the revolutionary and national liberation movement, progressive socio-political, scientific and other creative activities” [The Constitution of the USSR, 1977].

In other words, only political and ideological motives for immigration were recognized, which could serve as a basis for granting asylum. In the period after the collapse of the USSR, significant structural and legislative changes were carried out in the migration sphere, including the formation in 1992 of the Federal Migration Service. Serious problems remain, however, including endless systemic reorganizations and the regular adoption of contradictory legislation. Due to the difficulty of gaining a foothold and getting a job legally, immigrants who come to the country rarely use their potential effectively. Meanwhile, more than 43% of migrants who arrived in 2009 from the CIS and Georgia had vocational education, including 18.3% higher and incomplete higher education, and 24.8% secondary vocational education [Population of Russia, 2011]. Among temporary migrants with higher education 36.3% are ready to stay in Russia on a permanent basis, compared with an average of 27.1% among all migrants [Mukomel, 2013]. Thus, many migrants have both a qualification potential and a desire to integrate into Russian society.

In addition, immigration is also important as a factor in mitigating Russia's demographic problems: the country's population has declined from 148.3 million in 1991 to 146.5 million today and will continue to decline in the long term [Population of Russia, 2011; Demographic forecast…, 2016]. The situation is complicated by the fact that although migration is increasingly acquiring the nature of temporary and labor migration, replacing the permanent ethnically or politically determined in the first post-Soviet years, a significant proportion of labor migrants are illegal, or rather, irregular migrants. Estimates of their numbers vary significantly, from 2.1 million [Romodanovsky, 2012] to 3–5 million people. [The concept of the state…, 2012; Mukomel, 2005]. The expert consensus estimate is 2.4 million people. [Consensus assessment…, 2010], while the total number of labor migrants, including legally employed, varies from 3.8 to 6.7 million people. . Moreover, contrary to existing stereotypes, the demographic and labor resources of the post-Soviet countries, the main suppliers of labor migrants to Russia, are by no means endless. Their total number does not exceed 9–10 million people. [Population of Russia, 2011, p. 282]. In addition, a number of post-Soviet states, primarily Kazakhstan 1 and Belarus, are beginning to compete with Russia for certain categories of labor migrants, including skilled migrants. The irony here is that in the relatively near future, Russia may be forced to find alternative sources of labor, accepting migrants from countries with a much greater cultural distance from the host population than migrants from post-Soviet states.

Already today, for example, China is the largest supplier of labor migrants to Russia outside the post-Soviet space and the third overall after Tajikistan and Uzbekistan. In 2001–2009 The number of legally employed Chinese workers increased by 7 times. In 2011, the share of Chinese in the migration flow to Russia from the countries of the “old abroad” was 18.3% [Population of Russia, 2013, p. 481]. The largest number of Chinese in Russia was recorded in 2008 - 281.7 thousand. Their total number is now estimated at 200-600 thousand [Population of Russia, 2006, p. 346; Population of Russia, 2011, p. 278–279; Do you need ..., 2006]. The problem is that a significant number of migrations, especially labor migration, are irregular and therefore fall into a legal gray area; migrants themselves are unable to obtain legitimate documents or effectively protect their rights. Irregular migration is a phenomenon that goes far beyond Russia. Immigration usually focuses on ethnic aspects, the danger of "erosion" of national culture, the influx of illegal migrants, the growing pressure on the labor market and the mechanisms of social protection of the population, the growth of crime and corruption. With regard to emigration, the strongest emotions are caused, first of all, by the loss of intellectual and professional elites 2 - and in this regard, Russia is by no means an exception.

The modern world is characterized by a shocking asymmetry in the flows of intellectual and educational migration between the countries of the global South and the global North. and less than 1 thousand Americans. This shocking imbalance is giving rise to proposals to stop or at least slow down the brain drain using a number of methods. Among them are the introduction of restrictions on the departure of elite migrants from their countries, the requirement for them to return state expenses for their education and training, and in cases of temporary educational migration, the payment of a deposit (an apartment or other forms of real estate) when individuals receive state educational grants until they return home. The adoption of similar measures has been repeatedly proposed by a number of politicians and public figures in Russia. Migration on a global scale. The fact that the current migration situation is relatively new for Russia makes it very useful to study the experience of foreign countries that have faced similar problems in recent decades. These countries can be grouped into six main groups. First, these are the countries of traditional immigration, which include, in addition to the United States, Canada, Australia, New Zealand, Israel, and South Africa, which itself is now faced with a massive outflow of elite migrants and a large-scale influx of low-skilled irregular migrants from neighboring African countries.

Of particular interest to Russia is Israel's experience in attracting and adapting migrants, including those belonging to the elite categories. Secondly, these are countries that were previously the nuclei of multinational imperial formations (for example, Great Britain, France, Spain, Portugal, the Netherlands, Belgium, etc.), which, after their collapse, received significant flows of migrants of two main types. Initially, it was the return migration of representatives of the metropolis returning to their ethnic homeland (British, French, Turks, etc.), and then the migration of representatives of third world countries, primarily those who speak the languages ​​of the metropolises, are familiar with their culture and have the opportunity to rely on to support long-established ethnic diasporas of citizens of their former colonies. Thirdly, these are the countries of Central-Eastern and Southern Europe, as well as Russia, which are faced with a rapid change in their position in the global migration chain and the need to urgently form new migration structures, adopt legislative acts regulating migration and formulate the goals of migration policy.

Most of them simultaneously border both much more and much less developed countries. This feature of the geographical position culminates in the massive flow of refugees from the countries of the Middle East and North Africa (MENA), now passing through the territory of these countries and directed primarily to Germany and other most developed countries of Western Europe. Fourth, these are states that have experienced large-scale elite emigration in recent decades and are actively working to attract representatives of their foreign diaspora back to the country or other forms of cooperation with it. Among them are China, Republic of Korea, India, Taiwan. India, for example, today ranks first in the world in terms of the number of emigrants who have gone abroad, and their number for 1990-2013 was 100%. doubled 1. By the beginning of the current XXI century. 35,000 doctors and more than 300,000 computer scientists and specialists in the field worked in the USA alone high technology Indian origin. In the case of China, more than 2.6 million students completed their higher education abroad2, but 1.1 million (41.9%) returned home. Both India and China are actively working with members of their elite diasporas in the hope of stimulating a return home or developing interaction. India, in particular, introduced the so-called "overseas Indian citizenship" - an official document for those Indians and their descendants who permanently reside abroad. This document allows non-resident Indians to enter the country without visas, work as skilled professionals and hold certain property rights. China, in turn, began after 2010 to offer job guarantees and significant financial benefits to returning highly qualified specialists and Chinese students studying abroad. Fifth, these are states that have experienced explosive economic growth in recent decades and are forced to stimulate large-scale labor immigration, both elite and low-skilled.

Typical representatives are the countries of the Persian Gulf (however, Singapore and a number of other countries of Southeast Asia can be mentioned), for many of which the consequence of such a policy was the formation of deep divisions in society between local citizens and disenfranchised and often without legal status immigrants, often also different from the host population in terms of language and religion. Such situations are characterized in classical political science as "mutually reinforcing cleavages" - parallel and mutually reinforcing fractures in society. Belonging to one racial, ethnic, religious, linguistic, cultural, gender or social group implies belonging to a number of others in a similar situation, which dramatically increases social tension and the likelihood of conflict in society, turning areas with a high concentration of migrants into social “time bombs”. ". The events of recent years in Bahrain and a number of European countries, including France, highlight the danger of segregation of migrants and ignoring the need for their integration into host societies. In most of these countries, the situation is complicated by the authoritarian nature of political systems and the extreme weakness of civil society.

Meanwhile, in many of them, migrants have long been the majority of the population (in particular, in the UAE their share in the population is 84%, in Qatar - 74%, in Kuwait - 60%, and in Bahrain - 55%) . It should also be noted that a similar dynamics of the migration situation and a plurality of roles in the global migration network are also characteristic of the BRICS countries. Finally, sixth, it is important to study the experience of those countries that, while experiencing serious demographic problems, continue to restrain immigration flows for political and other reasons even at the cost of serious demographic and socio-economic losses - for example, Japan, and until recently the Republic of Korea.

It should be noted that other factors can cause similar consequences: in China, for example, such problems have arisen as a result of the long-term implementation of the “one family – one child” policy. The study of the social consequences of such a policy is especially important in the light of the widespread anti-immigrant campaigns unfolding in all three regions under consideration. European migration realities. With the end of the Cold War, attitudes towards migrants in general and refugees in particular began to change rapidly in the European West. Freedom of immigration from communist countries was one of the main Western slogans in the conditions of bloc confrontation: it was relatively easy to enter the West and receive refugee status. When, with the collapse of the communist bloc, and then the USSR, this issue lost its political significance and the exit doors opened, the West quickly lost interest in this problem, and in the fate of the majority of migrants.

Now he often sees them as a burden. The liberalization of the emigration regime is accompanied by a tightening of the screws in the West, which has made it difficult to enter the most attractive countries. In addition to the end of the block confrontation, internal difficulties in the EU countries also played a significant role - both economic (a slowdown in economic growth and the growing perception of migrants as competitors and in the labor market in the field of social benefits), and political (increasing interethnic tension and a series of conflicts in a number of host countries, and then the unfolding terrorist campaign, which led, in particular, to the rapid growth of the influence of right-wing nationalist parties). Meanwhile, the absolute number of migrants and their share in the population of the EU member states continued to grow. As of January 1, 2015, there were 19.8 million non-EU citizens (3.9% of its total population) living here, while the number of EU residents born outside of it was 34.4 million (6.8 %). 15.3 million citizens of EU member states lived in other states of the Union. The largest number of non-citizens is counted (million people) in Germany (7.5), Great Britain (5.4), Italy (5.0), Spain (4.5) and France (4.4). These five countries are home to 76% of non-EU citizens, while their share of the total EU population is 63%. In relative terms, the leader of the EU is Luxembourg, where immigrants make up 46% of the population. In six more countries (Cyprus, Latvia, Estonia, Austria, Ireland and Belgium), the share of non-citizens exceeds 10% 1. Their share is high in the population of Switzerland (27.4%) and Liechtenstein (63.7%), which are not members of the EU. With the intensification of economic difficulties and the growth of ethnic tension in many European countries, demands are being made for severe restrictions on immigration and reorientation of migration policy towards the preferential admission of highly qualified specialists to the detriment of all other categories of migrants, including refugees. In particular, former French President N. Sarkozy spoke about the need to move from "suffered" to "chosen" immigration. European experts distinguish between desirable (highly skilled labor) and undesirable migration. The first one singles out “inevitable” 2 and “involuntary” migrants – both those who enjoy the right to family reunification 3 and those who apply for asylum .

The implementation of plans to dramatically increase the share of “desirable” migrants could lead to serious structural changes in migration flows, negatively affecting the situation of low-skilled migrants and refugees. With the beginning of the formation of the Schengen area in 1990, the adoption of the Amsterdam Treaty in 1997 and the declaration in Tampere in 1999 of the goal of creating a zone of “freedom, security and justice”, the EU is increasingly oriented towards the parallel formation of two rigidly defined European migration regimes. Firstly, the guarantee of freedom of movement and the creation of a single labor market within the EU4. Secondly, the construction of high barriers around the European Union, cutting off undesirable categories of migrants, or at least complicating their entry into the EU. These regimes are often described as rights-oriented (intra-European) and security-oriented (external, prohibitive). They are also characterized as, respectively, "Europe without borders" and "European fortress".

The radical changes of the last two years, including the formation of a powerful flow of refugees from the MENA 1 countries, along with a series of terrorist attacks, have significantly complicated the European migration situation, sharply increasing popular support for far-right political parties actively using anti-immigrant rhetoric. Beyond the radical shift public opinion in many European countries to the right, the changing situation can lead to a serious deterioration in interethnic relations and the expansion of executive powers even in the leading democratic countries of Europe 2. The specifics of the North American situation. The American experience of migration policy is especially relevant and interesting, since the United States is the center of the largest migration system in the world: according to the US Census Bureau, in 2014 the number of immigrants in this country reached 42.4 million, or 13.3% of the population that at that time constituted 318.9 million 3 Only for 2013–2014 the number of immigrants increased by 1 million people. or 2.5%. The number of immigrants and their children born in the country today is 81 million people, which is 26% of the population. According to the Office of Immigration Statistics of the US Department of Homeland Security, in January 2012, there were 11.4 million irregular ("illegal") migrants in the country, 71% of whom came from Mexico and Central America. The centuries-old history of American immigration and the fact that representatives of one ethnic and religious group, Hispanic Hispanic Catholics, dominate in the incoming flow are also important, which increases fears about the erosion of the ethno-religious identity of the US population. This circumstance shows how the American migration situation is generally more favorable than in Russia and most other immigrant-receiving countries, since the cultural distance between the local population and the majority of immigrants - Christians who speak one of the main European languages ​​- is the smallest in the United States.

Immigration is the most important factor not only in quantitative growth, but also in qualitative changes in the composition of the US population. Today, its number continues to grow at a very high rate and, according to forecasts, by 2050 it will reach 438 million people. . Immigration accounts for approximately 1/3 of total population growth and is seen as a stimulant to economic activity, fueling both a low-paid workforce and highly skilled professionals, which means significant savings in training costs. Over 24 million immigrants are involved in economically active activities. It is interesting, however, that the US migration policy for a long time did not place much emphasis on the qualifications of migrants, giving priority to their ethnic origin and countries of origin in order to maintain a high proportion of Western Europeans among the population, primarily on the basis of discriminatory regional quotas, introduced at the beginning of the twentieth century.

Only after 1965, with the passage of the revolutionary Immigration Reform Act and a series of civil rights acts, did the US radically revise its immigration policy, opening its borders to third-worlders and skilled workers. Today, American immigration policy is aimed at achieving the following main goals: 1) ensuring a stable demographic supply of the US population; 2) maintaining its ethnic diversity; 3) provision of labor force of various categories 1; 4) admission of refugees for political, religious, ethnic and other humanitarian reasons; 5) stimulating the influx of highly qualified specialists, which reduces the burden on the US educational system, reduces the cost of training professional elites and brings significant income to American universities and the national budget (which, in particular, can be used to finance research and subsidize the education of students - American citizens) 2; 6) large-scale training of foreign students in American universities makes it possible to select the best personnel to provide them with work and residence in the United States and stimulate the formation of pro-American groups - carriers of a new political culture and ideology from those who subsequently return home.

Today, foreign students and immigrant employees in the United States make up about ½ of the academic staff in the field of natural sciences 1. The US migration policy, aimed at solving domestic socio-economic and political problems, is at the same time the most important foreign policy mechanism of "soft power". Recall that J. Nye characterizes it as “the ability to get what you want based on (your) attractiveness, and not through pressure or (monetary) payments” . Nevertheless, today the migration policy within the United States is subjected to harsh criticism. At the center of discussions is the issue of illegal immigration. The range of proposed solutions is extremely wide - from a complete amnesty for 11.4 million illegal migrants to their large-scale deportation. As a result of deep disagreements between supporters of opposite approaches in the migration sphere, the United States has not carried out deep structural reforms since 19863, and all presidents after R. Reagan “cracked their teeth” trying to propose their concepts of migration reform to Congress. Today, even the very expression "immigration amnesty" has become abusive in the American political lexicon. Carrying out a deep structural reform of migration policy was one of the most important pre-election promises of B. Obama in 2008, which provided him with impressive support from Hispanic voters (about 10% of the electorate). The total number in the United States of the so-called "Hispanics" - people with Hispanic roots - reached 55.3 million people in 2014. (17.3% of the country's population) 3. Now they make up 46% of emigrants in the United States (19.4 million people). During the 2012 presidential election, B. Obama received 71% of the Hispanic vote (the best result for a Democratic candidate since 1996), while Republican M. Romney received only 27%. This situation is forcing the Republican elite to frantically seek compromise solutions, supporting some aspects of migration reform in the hope of returning at least part of the "Hispanic" electorate. However, the fact that, contrary to his campaign promises, during his presidency, B. Obama never proposed a comprehensive immigration reform project to Congress, led to a sharp decline in the turnout of Hispanic voters in the 2014 elections, becoming one of the most important reasons for the crushing the defeat of the Democratic Party on them.

B. Obama's immigration concept includes both liberal provisions and rather strict restrictive elements. universities or military personnel. The concept of immigration reform also provides for the liberalization of the visa regime for highly skilled workers and entrepreneurs who are ready to invest in the US economy. The defeat of the Democrats in the 2014 elections further weakened the chances of passage of new immigration legislation by Congress, which could lead to the preparation of executive acts by the president, complicating his relationship with the legislature. On the whole, H. Clinton supports the concept of B. Obama's immigration reform, offering even more opportunities to legalize certain groups of illegal migrants, primarily young people and people with a high level of education. The priorities of migration policy cause serious controversy even among the supporters of the liberal approach. Thus, there are tensions between those who support the general lifting of restrictions on immigration and the mass legalization of illegal migrants, and those who propose a radical increase in the quota of highly qualified specialists, while at the same time a sharp reduction in the number of persons arriving through family reunification.

In particular, opponents of the presidential approach warn that large-scale legalization of illegal immigrants could create an unfavorable precedent, stimulating further irregular migration, including "reproductive tourism" of pregnant women 3. In their opinion, this situation is unfair to those who came to the United States legally , but sometimes have to wait decades to obtain a permanent residence permit or citizenship. The consistent decline in Hispanic support for Republicans is forcing the party leadership to seek compromise in order to support certain aspects of immigration reform in order to win the support of at least part of Hispanic voters. Such attempts are met with fierce resistance from conservative party activists, leading to a deep split within the party. Trump's candidacy as the official GOP candidate in the 2016 presidential election, with his openly anti-immigrant populist rhetoric, including promises to build a wall along the border with Mexico, deport all illegal migrants, and temporarily block Muslim immigration, confirms this intra-party split and proximity of public opinion trends in the US and Western Europe. They are very representative in terms of the growing uncertainty of the white majority in maintaining their positions in American society and fears for their future, especially during the years of B. Obama's presidency. Indeed, in this presidential cycle, the American electorate appears to be fractured along ethnic, racial, gender, religious, and sexual orientation lines. D. Trump is supported primarily by representatives of the traditional white middle class, especially men, while H. Clinton is supported by a broad coalition of minorities (African Americans, Hispanics, Muslims, feminists and women in general, as well as sexual minorities). Before the 2016 elections, more than 90% of African Americans supported H. Clinton, and D. Trump - only 2-6%. In July 2016, H. Clinton was supported by 66% of Hispanic citizens, D. Trump - by 24%. In such a situation, any outcome of the election can have very serious consequences for the internal stability of the United States: the losing side, this time, may perceive an unfavorable result as an open attack on their interests. And yet, unlike Europe, the American electorate before the start of the current election campaign did not perceive the problem of immigration as one of the most important for the country. Getting this issue to the center of the political agenda and presidential debates turned out to be directly related to the emergence of D. Trump's candidacy and his aggressive campaign, which seriously reshaped the priorities of voters. This fact highlights the destructive potential of both populist political leaders and sensational and ideologically charged sources of mass media in terms of radicalizing public opinion and worsening interethnic and interfaith relations in the country.

In particular, the ten-point migration program, announced by D. Trump on August 31, 2016, proposes a significant tightening of the US migration policy, which can have very serious consequences both for American and for interracial and other intergroup relations in this country. Its implementation can lead to a sharp increase in government control over the sphere of public relations and the personal lives of American citizens, while D. Trump's policy towards the Hispanic and Muslim communities can lead to a deterioration in relations with a number of interstate organizations and foreign states, including Mexico and Latin America in general, as well as many Muslim countries. At least at present, D. Trump is ready to accept only highly skilled migrants, blocking the immigration of representatives of almost all other groups. Unfortunately, in this sense, current trends in the US migration sphere, including the orientation of public opinion, since D. Trump's position reflects the views of a significant part of American voters, are increasingly beginning to resemble the situation in the EU. Many are now calling for the United States to adopt the Canadian points system as the main principle for shaping the migration flow and assessing the "quality" of potential immigrants. permanent residence permits are issued on the basis of qualifications (in the US - only 13%). Meanwhile, in Canada, this system is also being criticized, both because it often ignores the humanitarian aspects of immigration, and because it does not effectively take into account the needs of the labor market.

In particular, the governments of a number of Canadian provinces emphasize that professions are the most scarce now. middle category complexities, including, for example, middle and junior medical personnel, while the points system gives an advantage to the most qualified and expensive personnel. Probably the optimal solution lies somewhere between the Canadian and American varieties of immigration policy. In general, the experience of both the United States and Canada speaks of the significant benefits of immigration and the ability of the state and society to integrate large masses of migrants without endangering the foundations of democracy. Instead of a conclusion - foreign migration experience and Russia.

The migration situation in Russia is characterized by noticeable parallels with other leading countries of immigration, and therefore the study of foreign practice in this area and the application of its positive aspects to Russian reality would be very useful. Despite numerous problems, all world experience testifies to the ability of the state and society to accept and integrate large masses of migrants without endangering the foundations of democratic power. However, the same experience simultaneously shows that the policy of segregation of migrants, which creates ethnic enclaves and drives them into legal gray areas, can have negative consequences. The specificity of the Russian situation is associated primarily with the relative novelty of the problem of large-scale immigration, as well as the institutional and psychological unpreparedness of the state and society for the influx of a large number of migrants of other cultures. Moreover, the current situation in Russia is complicated by a number of points. First, not only by the presence of millions of unentitled migrants who differ from the local population in ethnic, religious and cultural terms, but also by the weakness and inefficiency of the government structures that should work with them. Second, Russia today stands out for its lack of large-scale anti-immigrant parties and movements and charismatic populist leaders. However, this feature of the current Russian political situation is largely due to the fact that the Russian state itself is consistently engaged in discrimination against certain groups of migrants, preventing the legalization of irregular migrants.

Thirdly, the introduction in recent years of certain restrictions on the activities of foreign and international non-governmental organizations, which all over the world play a crucial role in helping migrants and refugees. This not only reduces the range of services provided to migrants, but also creates an additional burden on the budget and government agencies. Fourth, the relatively modest role of church organizations is palpable. Meanwhile, in the United States, for example, churches play a crucial role in helping migrants and refugees, seeing them as potential parishioners. In Russia, both the state and the population are very skeptical about the problems of migrants in general and refugees in particular. This circumstance, along with the weakness of civil society, xenophobic sentiments widespread among the population, in the state apparatus and in the press, prevents proper understanding of the seriousness and importance of migration issues, overshadows both the strategic and human components of migration problems, reducing the effectiveness of policy in this area. . Meanwhile, an effective migration policy could contribute to the solution of many pressing Russian problems in the socio-economic and demographic spheres. No less important are the external aspects of migration policy: immigration from neighboring countries is the most important mechanism for stabilizing the socio-economic and political situation in border regions based on such factors as migrant remittances 1, reducing the demographic burden and easing pressure on social services and labor markets in countries of emigration, providing migrants with educational services and immersing them in Russian culture. The use of these mechanisms could significantly expand the potential of Russia's "soft power".

In this article, we would like to systematize our experience in conducting data migration in large corporate projects related to the transition of Customers to work in 1C:Enterprise 8 configurations.

At the same time, the main emphasis in the article will be placed, first of all, on the technological component of the migration process. The organizational component is also affected, but to a lesser extent.

Terms and Definitions

Data migration is commonly understood as a final sequence of work, a project aimed at a one-time mass movement of data from source systems (historical systems) to a receiver system. At the same time, the operation of this data in the source systems is terminated.

Data migration should be distinguished from data integration. Integration, unlike migration, is a permanent part of the IT architecture, and is responsible for the flow of data between various systems and data warehouses - and is a process, not a project activity.

The migration scheme in general looks like this:

Rice. one

Historical systems- databases of the Customer's company, which are planned to be completely or partially replaced during the implementation of the new system.

Receiver system- target system, arbitrary configuration "1C:Enterprise 8".

Initial data- data downloaded from historical systems into an arbitrary xls-file format. In this case, the xls format seems to be one of the most convenient, since the ability to upload to an xls file is present in many "previous generation" accounting systems.

As a modern alternative, it is possible to consider the format of xml files as a transport.

There are also options for using an intermediate database.

Transformation, conversion- the process of converting raw data into data for download. Data transformation occurs in accordance with the templates for loading. The result of the transformation is the data to be loaded.

Data to download- data to be loaded into the receiving system. In this article, as well as the source data, the xls format is considered.

Download Data Templates- description of the data tables to be loaded into the target system.

Stages of migration

Consider the process of preparing and conducting migration step by step.

The organizational stages of migration include the following items:

· Define the migration strategy. At this stage, the Contractor and the Customer agree on the technology of migration work;

· Determination of the composition of the working group on migration. The working group should include specialists from both the Contractor and the Customer who are sufficiently familiar with the operation of historical systems (on the part of the Customer) and the target system (on the part of the Contractor);

· Preliminary migration plan. The migration plan will be repeatedly adjusted during the course of the project;

· Date periods for unloading data from historical systems, data volumes. Data cutoff periods for migrations, dates of test and final migrations. This information can be attributed to the migration plan;

· Composition of data to be migrated. Reference data, classifiers, transactional data, balances, turnovers, etc.;

· Issues of checking the quality, correctness and integrity of data during and after migration;

· Issues of rollback to the previous state in case of failures.

Let us dwell in more detail on the technological stages of migration.

Rice. 2

1.Preparation of data loading templates

The data load template contains technical descriptions data tables to load, algorithms and loading rules for the current template.

Each template is generally intended for one or more related tables on the destination target system.

The template specifies:

Description of all fields of the xls data file to be loaded, including:

o Field name

o Mandatory field sign

o Field Filling Example

o Note

Description of the rules for loading the table of the target system based on the data to be loaded (order in the case of several related tables, search algorithms for key fields, etc.)

· Description of filling in the fields of tables of the target system directly in case something other than one-to-one data transfer from the data file for loading is envisaged. Relevant for reference fields, for example.

In the course of work on this stage, the Contractor must also prepare a data file loader for loading. In the case of working with xls files, this task is not particularly difficult.

2. Identification of data sources

This step may start together with the previous step “1. Preparation of data loading templates”.

As part of this stage, the Customer's specialists determine from which systems and what data can be uploaded. It is also necessary to determine what data maybe may be needed.

As a rule, in large migration projects, the identification of a complete exhaustive list of data sources can take quite a long time and occurs as work is carried out at subsequent stages.

There are often situations when, in order to ensure the integrity of information in the future, some data has to be transferred from printed sources (digitized) or even entered into tables from the words of key employees of the Customer.

However, at this stage, you should try to identify as much of the necessary data as possible.

3.Unloading initial data

The process of unloading data from historical systems may take a sufficient amount of time, especially if there are many systems, they are different and different departments of the Customer are responsible for them. Should be considered this moment during test and final migrations.

The most convenient option seems to be uploading to xls files. Many older IT systems support this option.

There may also be options for uploading to csv format, dbf , xml formats and others.

It should be noted that for one reason or another (security issues, for example) the Customer cannot always provide the data upload in full at this stage! Just a data structure and a few test positions. Thus, a situation may arise that during test and final loads, poor-quality data will be found in the source tables, which will lead to unplanned errors.

To minimize this problem, the volumes of test downloads from historical systems should be specified in advance.

4. Data mapping

Mapping (data mapping) - in the general case, the process of comparing data from historical systems and the receiving system. That is, the source data and the data to be loaded.

The mapping stage is the most time-consuming stage and can take more than 50% of all work on the migration task.

At this stage, the entire working group of the project on migration is fully involved.

In the process of data mapping, it is necessary to distinguish the sub-stages of table mapping and field mapping.

· Mapping of tables, or mapping of templates - comparison of tables of initial data and data templates for loading. The correspondence can be either 1:1 or N :N . As a result of this work, a register of table mapping is compiled and maintained. This sub-step is required for the next field mapping sub-step and to keep track of the overall mapping progress.

Template group 1C

Template name 1C

File name-

source

Rules for generating a source file

Responsible

Status

Note

NSI

Sample_

Nomenclature

Nomenk

latura.xls

In system N set selection
. Save to txt
. Open in xls, columns - text
. The first line is the header
. Number of columns - 15
. Check number of lines in txt and xls
. Sheet name is always "Sheet1"

Ivanov I.I.

in work

· Mapping of fields - comparison of fields of tables within already defined mapping of tables. The result of this work is a register of field mapping.

No. pp

Cl. field

Mandatory

Template field name 1C "Template_Nomenclature"

Description

Field name "Nomenclature.xls"

Fill algorithm

The code

Directory element code

The code

Name

Name

Yes

This group

Contains one of the following values:
. 1 - for groups
. 0 - for elements

If code length=11 characters and last 4 characters<>"0000", then this element is "0", otherwise the group is "1".

Full name

Directory element name

Name

If ThisGroup =1 , Then "", ElseIf ThisGroup=0, then Name.

As part of this stage, possible work on data normalization should also be carried out.

5. Preparation of transformation rules

Unlike the previous stages, this stage is technical and involves the work of the Contractor's developer.

Based on the agreed field mapping registries, the Contractor's specialists develop data transformation rules.

For operational work during the preparatory migration stages and beyond, during test and final migrations, it is important that there is a convenient environment for developing data transformation rules (scripts) and an environment for converting initial data into data for loading.

The requirements for this environment include:

· Convenience and speed of development of transformation rules;

· Data conversion speed. Input and output files can be hundreds of thousands of lines long!

Ability to work with multiple input files simultaneously;

· Ability to save transformation rules in separate files.

For our migration projects, we have developed a specialized developer's workstation, based on the standard processing of the "Request Console" 1C.

The "Query Console" handling has been improved to allow direct requests to xls files.

Here is an example of combining two source xls files Staff.xls


Employee code

Surname

Name

middle name

Date of Birth

2423

Ivanov

Ivan

Ivanovich

17.11.1992

1523

Petrov

Basil

Aleksandrovich

04.02.1991

4363

Sidorov

Kirill

Nikolaevich

01.05.1995

Denisov

Denis

Denisovich

01.01.1990

And Operations.xls with pages:

Write-offs

Employee code

date of

Sum

2423

01.02.2014

1523

02.02.2014

4363

03.02.2014

04.02.2014

100000

2423

05.02.2014

1523

06.02.2014

4363

07.02.2014

2356

08.02.2014

140000

2423

09.02.2014

1523

10.02.2014

4363

11.02.2014

23523

12.02.2014

80000

And Income:

Employee code

date of

Sum

01.05.2004

02.05.2004

03.05.2004

04.05.2004

2423Date of Birth

Amount received

Write-off amount

Ivanov Ivan Ivanovich

2423

17.11.1992

1341234

1010

Petrov Vasily Alexandrovich

1523

04.02.1991

245245

Denisov Denis Denisovich

01.01.1990

380000

320000

Sidorov Kirill Nikolaevich

4363

01.05.1995

613382

26336

TOTAL:

2579861

347842

Note that the example is artificial, specially selected to demonstrate all possible stages of data source transformation.

The technological sequence of transformation operations here is as follows:

Using the Access SQL query language (which provides significant additional features compared to the 1C query language), an initial query is created that retrieves data from the xls file into the 1C environment. At the same time, various checks and normalization of data are already possible at this stage.

ADO data access technology provides high speed.

Rice. 3

2. Request in 1C language - the main request that implements the field mapping algorithm. And also: enriching the downloaded data with data from the 1C database, regrouping, merging with the results of queries to other source xls files, etc.

3. Post-processing of the result of the 1C request, if necessary. It is implemented using a script in 1C language.

For example, the addition of the line "TOTAL" for the columns of amounts is implemented here.

4.Recording the final data set to xls-file.

In the general case, at the output we get the final files for uploading to the target 1C database.

Also, this tool allows you to save data conversion rules in a separate xml file:

In addition, it is possible to work in batch mode , which is especially important with a large amount of heterogeneous migrating data.

During the previous stages, the preparatory part of the work as a whole ends - all data sources are identified, the source data is unloaded from the sources, templates for loading into the target database are prepared, data mapping is prepared and, finally, data transformation scripts are developed.

It should be noted that before the final migration, you should definitely conduct several test ones. During test migrations, the Contractor together with the Customers identify:

conversion errors, data loading errors

carry out a preliminary assessment of the quality of data loaded into the target system

Based on the results of test migrations, they draw up / update the final migration plan

7.Data verification

Checking the quality of the downloaded data should be carried out both after the test migrations and at the end of the final migration. During the reconciliation, the following indicators can be checked:

· Coincidence of totals for balances, for documents;

Quantitative matches, such as the number of OS;

· Correctness of filling of separate selective entities;

Please note that certain checks of migrating data, issues of data normalization must be addressed throughout all migration processes. You must always ask yourself what needs to be done at the current stage in order to avoid mistakes in subsequent stages.

For example:

· Check for duplicates by key fields. It is possible and necessary to carry out still on the initial data;

· Casting field types;

· Referential integrity;

· Mathematical inconsistencies. For example, checking for empty numeric fields, which are planned to be divided during transformation;

· In general, checking the mandatory filling of fields;

· Replacing incorrect characters. For example, English characters in Cyrillic fields ("o", "a", "e", etc.) This is especially true for key fields!

Checking the values ​​of string fields for compliance with the types of the receiving system (Length restrictions)

After the final migration is completed, according to a predetermined migration strategy and migration plan, a decision is made on the further operation of the legacy systems.

Often, operation is completed immediately after the final reconciliation of data and the success of the migration - users of the new system no longer keep records in parallel in two systems, but completely switch to the new system. At the same time, access to the old system is preserved in read mode.

In some cases, parallel operation of two systems may occur for the period of trial operation (OE) and even more than this period. The issue of parallel work of users in two systems is closely related to the issue of the possibility of a rollback to the old system, if the migration (or, in general, the operation of the new system!) Is found to be unsatisfactory.

Conclusion

In conclusion, I would like to note that when it comes to migrating large transactional systems, which include many 1C:Enterprise configurations, the transition to a new system can be very time-consuming.

Therefore, it should be remembered that any such project requires careful preparation and must be accompanied by an individual plan. However, regardless of the type of systems being migrated, the volume of databases, etc., the general scheme of migration looks almost identical.

The process of data migration can be one of the stages of the IP implementation project, or it can be organized as a separate project. Under the data migration process in the framework of this work, we mean design work that covers the full cycle of tasks related to data migration: from planning data migration work to evaluating the results of the data migration stage.

In any case, the data migration process is divided into several interrelated successive stages; in this study, all steps of the migration process according to the Oracle and IBM methodology will be sequentially considered.

The life cycle of the migration process begins after the formation of a strategy and risk assessment of the data migration stage. The flow diagram of the migration process is presented in the process diagram.

The goal of any data migration process is to map the information, data types and data formats of the old system to the data types and data formats of the new system. During data migration, the “Data Extraction” stage corresponds to the selection and unloading of data from the old system, and the “Data Loading” stage corresponds to the transfer of the received data from the old system and their loading into the new system. The migration process will be discussed in more detail below.

After the data migration planning phase is completed, the next step is to determine the requirements for the migrated data. This stage includes the development of customer requirements and their description in the relevant project documents. During the Gather Requirements phase, the responsible role within the project team for the deliverable of the phase is the Business Analyst or System Analyst. This stage of migration will be covered in more detail in the third chapter of this work. The output information of the stage of determining data requirements for migration is a description of the structure and composition of the data for migration.

The stage of collecting data requirements for migration, as a rule, is very closely related to the next stage - the development of algorithms for transferring data from the source system to the target system. During the design phase, detailed specifications are created by analysts describing the data types of the source system and their relationship to the data types of the target system. Such specifications describe the data structure for migration, their volume, source, destination. The specification is a source for setting tasks for the developer, who will design and develop specialized software for data transfer. At the design stage, an analysis of the existing data architecture in the source system is carried out - analysis "as is" and development of data architecture in the target system - "to be". When analyzing the existing data architecture, all limitations and IT infrastructure are identified and taken into account, as well as their impact on the operation of the target system with migrated data. The output artifacts of data architecture analysis can be documents such as logical data models (ER diagrams, database models), dictionaries and reference books with a detailed description of each element and its attributes, a description of business rules for working with data, information about systems interacting with source system for information exchange and integration.

The results of the collection of requirements and design are the basis for choosing a method and determining the technology for data migration. Migration can be carried out offline or online, the categorization of methods depends on whether the applications are maintained during the migration process. The choice of migration method and means is determined by a combination of factors, including available system downtime, business dependence on partners, data volume, physical location of source system data stores, information security policy of the source system and target system.

The stages of analysis and planning described above can be combined into a general preparatory stage. The developed procedures and migration mechanisms regulate the stages of extracting, transferring and loading data into a new system, that is, all steps of the ETL process are carried out sequentially. After receiving the data necessary for migration, the phase of loading this data into the target system begins, before the start of which it is necessary to single out a separate stage - verification of the migrated content.

Checking the compliance of the uploaded data with the requirements can take place online - directly at the entrance to the target information system or offline - as an intermediate step in the migration process. Upon completion of loading data into the target system, an additional check is performed, often both systems are launched for parallel operation. Test activities of parallel work are planned when designing the rules and procedures of the migration process. Within the framework of the migration process, the parallel operation of the two systems can be considered as trial operation. The result of trial operation can be a confirmation of the full operability of a new system with migrated data. If massive errors are detected during the parallel operation of the source system and the target system, a decision can be made to re-migration the data and reload the content. The agreed migration results are recorded in the target system trial operation log with loaded data, test cases performed, questionnaires can be compiled to check the compliance of the migrated data with the requirements of the target system.

Testing activities are not limited to the parallel operation of the source system and the target system. Tests can be run on samples of migrated data to detect errors early and fix them before development of the migration software begins. Fixing errors earlier saves budget and avoids re-downloads of data. Testing activities may include data audit activities during the migration process. Data audit allows you to track the state of data and avoid errors caused by changes in content that may be made by users already during the migration work.

After agreeing on the migration results, the stage of post-migration work starts, including checking, cleaning and testing the performance of the target system, in general, after data migration. Cleaning can be done manually or with software tools. Data cleaning is performed to remove obsolete information and meet the information requirements of the new system.

The data migration methodology given above assumes that the most “bottleneck” in organizing this stage of the project is the stage of planning and working with the customer’s business requirements, that is, collecting requirements and designing, so we will consider approaches to solving the problems of these stages in more detail in the following parts of the work. In addition to the stages of planning and developing business requirements, special attention should be paid to the stage of evaluating the results of the work of the data migration stage, since in accordance with the Deming cycle (PDCA), it is the performance of work assessment activities that is a condition for the success of similar work in similar projects.

1.1. Features of data migration planning

Data migration planning is the first step in the process life cycle and is done by understanding the key risks of the process and the migration strategy. In addition to the migration strategy, input information can be a section of the terms of reference or a document on the scope of the entire project, dedicated to data migration. At the planning stage, the scope of the data migration process is determined, the goals of the data migration process that are achievable under the conditions of the design constraints (data sources, top-level requirements) are established. To define the scope of the migration process, it makes sense to involve business users who have an understanding of how the system has handled data in the past and how it should handle it in the future. Further, depending on the migration method, a deadline is set, and the necessary resources are allocated within the specified budget. When planning data migration, an important point is to identify the participants in the process from the customer's side, that is, those business users and technical specialists of the customer who are responsible for data management. The following design artifacts can be generated at the output of the data migration process planning process:

  • - Document on the framework of data migration;
  • - Work plan for data migration indicating the responsible members of the project team;
  • - Communication plan during the migration phase.

The organization of the migration stage in IP implementation projects begins with the planning stage, where it is necessary to draw up a work plan, calculate the necessary resources and deadlines.

Work packages at the migration stage should correspond to the phases of the process life cycle, an approximate structure of the work schedule can be as follows:

  • - Planning and designation of data migration framework;
  • - Business analysis and documentation of requirements;
  • - Selection, customization or design and development of specialized software;
  • - Data transfer;
  • - Validation of migrated data;
  • - Experimental operation;
  • - Post-migration cleaning and testing work;
  • - Coordination of the migration results, evaluation and closing of the stage of the implementation project.

The assignment of responsible members of the project team for the implementation of the work packages of the data migration stage occurs at the planning stage after the work plan has been drawn up.

The selected project roles are assigned clusters - areas of responsibility defined in the MSF methodology. It is worth noting separately that by product management in the context of migration we mean managing the quality of migrated data and the health of the target system after migration. Release management in terms of the migration process - performing iterations of the migration process, receiving and loading migration data.

In accordance with the MSF model, the following distribution of responsibilities between role clusters is assumed:

  • - System Analyst - program management, customer satisfaction;
  • - Development manager - program management, product management, release management;
  • - Developer - development of algorithms or specialized software for transferring data to the target System, specialized software (if necessary);
  • - Tester - testing, release management.

In order to clearly demonstrate the participation of the involved human resources in the activities of the data migration process, we will compile the RACI matrix - given in Appendix 1 to the work (see Appendix 1 - RACI Matrix for data migration work).

It should be noted that the development manager (technical manager) is considered as the leader of the data migration team, so he is responsible for conducting the entire process as a whole. However, if data migration is carried out as part of a large-scale IS implementation project, where the manager of the entire project is appointed, then the technical manager of the migration stage will only be an executor in the tasks related to determining the timing and recruitment. Decisions on personnel, resources and deadlines in this case are taken collectively by the project management.

The service for migration of the existing IT service management automation system from the HP OpenView Service Desk 4.5/5.x product is aimed at solving the following tasks:

  • reducing operational risks by switching to a product that is fully supported by the developer and his partners in the Russian Federation;
  • reducing migration risks, reducing the cost and timing of migration by switching to a solution that is ideologically as close as possible to HP OpenView Service Desk 4.5 / 5.x (does not require serious reworking of existing processes and complete retraining of personnel);
  • ensuring the possibility of further development of IT management practices by overcoming the functional limitations inherent in the current automation system.

Migration is performed to a specialized design solution OMNITRACKER CleverENGINE, implemented on the basis of the OMNITRACKER platform. This solution provides customers with the following important benefits:

  • the CleverENGINE solution was originally created with the need to be as close as possible to the HP OpenView Service Desk product to ensure a "soft" migration. CleverENGINE has a similar data model and the same organization principle user interface. This allows you to save effort and money during migration for the forced adjustment of existing management processes, retraining of personnel, and also leaves the possibility of transferring historical data;
  • CleverENGINE was designed and built by existing IT management consultants. The functionality built into CleverENGINE is far superior to HP OpenView Service Desk. In particular, it provides visualization of the CMDB, accounting and control of the use of software licenses, release management, maintaining a knowledge base, automating routine operations, a powerful approval mechanism, and much more. This allows customers to improve their IT management practices by overcoming the functional limitations of their current automation system;
  • the OMNITRACKER platform is an environment for the rapid development of various workflow applications (including those outside the scope of IT service management). Solutions based on OMNITRACKER, as a rule, are transferred to customers in an open form - available for adaptation and further development. Many customers develop their own automation systems on their own with little or no help from partners. At the same time, on the basis of one platform, it is possible to gradually develop the functionality traditionally inherent in various products - service management, project management, document management and other solutions.

As part of the migration, a survey of the existing automation system is carried out, and introductory seminars on CleverENGINE are held for key personnel of the customer. Based on the results, the requirements for the final solution are specified, opportunities are identified to get quick benefits through the use of ready-made CleverENGINE functionality that was not available before. Then a new automation system is built, qualification tests are carried out, and personnel is trained. Further, if necessary, the transfer of historical data is carried out and a switch to a new automation system is performed. After that, the customer is provided with technical support for the period of trial operation. At the end of the project, the automation system is transferred to commercial operation.

results

As a result of the migration project, the customer receives:

  • a new automation system with the preservation of the management processes that operated in the old automation system;
  • new functionality of the automation system, previously unavailable;
  • a set of Russian-language user documentation for the new automation system.

Typical project profile

The duration of the migration project is from 2.5 to 4.5 months - depending on the degree of "customization" of the existing system based on HP OpenView Service Desk (the number of business logic rules, forms and data views, reporting, integration solutions, and so on ), as well as the need to transfer historical data. The project is carried out by 1-2 specialists. Specialists with expert knowledge of both products - both HP OpenView Service Desk and OMNITRACKER CleverENGINE - are involved in the work.

The service for migration of the existing IT service management automation system can be performed for systems implemented on the basis of various software products. Depending on which product is migrated from, the customer will receive different benefits, the project will require different resources. At the same time, the general logic of the organization of work is preserved.

Examples of completed projects

Examples of projects completed by Cleverics consultants:

  • Migration of the ITSM process automation system from HP OpenView Service Desk to OMNITRACKER CleverENGINE and optimization of user support activities (SK Alliance, 2014)
  • Migration of ITSM process automation system from HP OpenView Service Desk 4.5 to OMNITRACKER CleverENGINE (KATREN, 2013)
  • Migration of ITSM process automation system from HP OpenView Service Desk 4.5 to OMNITRACKER CleverENGINE (VTB24, 2012-2013)
  • Migration of the ITSM process automation system (AutoSpetsCentre, 2010)


Loading...
Top