In the era of digitalization, personal data has become as valuable as gold. Companies of all sizes collect information about their customers, partners, and employees, facing challenges related to their proper storage and protection. So how can we ensure the security of these valuable data? Check Managing and Storing Personal Data.
When do we process personal data?
Every day, we process personal data at work. Often unknowingly. An example is using a CRM system. We enter customer information there. This happens when signing contracts or registering orders. Such data can include names and addresses. Also, phone numbers and emails. Sharing data in emails is another daily practice. Sending a colleague a list is an example. It might be for a training session. Or a report with customer survey results. This way, we share personal data.
Processes like verifying an employee’s identity exist. This happens when booking business travel. Or updating contact information in HR. Scanning business cards is another example. These might be from conferences. They go into an electronic database. These are times when we process personal data. It’s crucial to know the data’s nature. And to maintain protection standards.
Where can I find personal data?
We store much information on company laptops. This includes email correspondence and key documents. Often, we store sensitive data unknowingly. An example is a spreadsheet with customer contacts. Or an annual summary of employee salaries.
While cloud services, such as Dropbox or Google Drive, simplify storage and file sharing, they come with certain risks. Despite advanced security options available, users sometimes neglect to use strong passwords for their accounts. Also, smartphones, which have become an essential tool at work, are a source of a vast amount of data – from emails to specialized company apps.
How easy is it to share personal data?
Not every data leak is a result of malicious intent. In fact, human errors are one of the main culprits of unauthorized data sharing. Imagine accidentally sending an email with an important attachment to someone who should never see it. Or carelessly sharing an internal company document with all employees instead of a limited group. Such oversights can lead to serious consequences for both the company and the individuals whose data was disclosed.
Furthermore, there are many less obvious, but equally risky situations that can lead to data loss. Using open Wi-Fi networks, such as those in cafes or airports, is undoubtedly convenient but also poses risks. Hacker attacks, eavesdropping, or attempts to break into our device are much more likely in such places. Even one careless connection can result in the theft of valuable information.
Contact us
Facing constant challenges related to the protection and management of personal data, we found the answer in a product from Wizards – Oblivio. For us, it’s not just an advanced tool, but above all, a guarantee of the security of our data. Oblivio integrates seamlessly with many systems in our organization, making the data processing simpler and more effective.
In a world where information has become the most valuable resource, thanks to Oblivio, we are confident in the proper management and protection of our data. Regardless of the scale of our operations, Oblivio provides us with tools tailored to our needs while ensuring the highest security standard.
Instead of constantly worrying about security and compliance, we’ve trusted the proven solution offered by Wizards. With Oblivio, we are sure that the data of our customers, colleagues, and partners are protected in the best possible way.
The General Data Protection Regulation (GDPR) was introduced by the European Union to address growing concerns about privacy and data protection in a world increasingly dominated by technology. The implementation of these regulations has revolutionized the way businesses collect, process, and store data. But what exactly do these changes mean for companies operating in Europe? And what are the consequences of not adhering to them?
Portuguese Hospital Fined by CNPD
In 2018, the Comissão Nacional de Protecção de Dados (CNPD) fined Barreiro-Montijo Hospital 400,000 euros. This Portuguese authority oversees personal data protection. The incident is a major example of GDPR enforcement in the European Union.
The Barreiro-Montijo Hospital Center in Portugal faced penalties for numerous and grave violations. An inspection revealed that the hospital lacked internal regulations for creating accounts and managing access to medical data. Moreover, there was a failure in taking steps to remove accounts of employees who had left the hospital. Additionally, patient data access was mishandled, leading to breaches.
Permission Management: How to Avoid Similar Issues?
Modern institutions, particularly those in the medical sector, must give special attention to permission management. Adequately structured and consistently implemented procedures in this field can not only shield institutions from potential sanctions but also elevate the overall level of information security.
Introducing a clear security policy is the cornerstone. It meticulously dictates who can access information and how much. Yet, that’s just the beginning.
These procedures should encompass not only the granting and revoking of permissions but also their regular reviews and updates. As the organizational structure changes and the roles of employees shift, permissions should be readjusted to continually reflect actual needs and maintain optimal security levels.
Moreover, it’s prudent to introduce systems monitoring access and user activities. Such systems not only bolster security by detecting unauthorized access but also act as audit tools, potentially providing evidence in the event of violations.
In conclusion, training staff on security policies and permission management is vital. Even the most robust system can falter if employees lack awareness of their roles and responsibilities in data protection.
Conclusion
The penalty given to Barreiro-Montijo Hospital in Portugal is a warning. It highlights the importance of permission management. It also emphasizes the consequences of GDPR violations. Safeguarding personal data has multiple purposes. It’s not just about avoiding fiscal penalties. It’s mainly about building trust with customers and patients. Institutions hold many people’s data. In today’s era, data protection is essential. It’s not a luxury.
The database management system (DBMS) is a critical component of many IT systems. This article will compare five popular DBMS: MySQL, DB2, SQL Server, Oracle, and PostgreSQL. By understanding their unique features and applications, we can make an informed decision about which DBMS best suits our needs.
MySQL
MySQL is an open-source relational database management system (RDBMS) that was first introduced in 1995. It is known for its simplicity and ease of use, making it a popular choice for web-based applications, especially those developed in PHP. MySQL offers support for a wide variety of data types and SQL (Structured Query Language) for managing data.
DB2
IBM developed DB2, a hybrid database management system combining relational and object-oriented features. Launched in 1983, DB2 is celebrated for its reliability and performance, ideal for enterprise-level applications. It offers a variety of features, including support for various SQL standards, stored procedures, triggers, and multiple types of indexes.
SQL Server
Microsoft’s SQL Server is a relational database management system introduced to the market in 1989. Particularly in corporate environments that utilize other Microsoft products, like .NET, its prevalence is notable. This platform offers a range of advanced features, including support for SQL queries, stored procedures, functions, triggers, and various types of indexes. Moreover, it has strong support for various data types, including JSON and XML.
Oracle
Oracle Database, often simply referred to as Oracle, is one of the oldest and most advanced relational database management systems. It was introduced to the market in 1979 by Oracle Corporation. Oracle is known for its reliability, scalability, and advanced features like support for PL/SQL, a stored procedure language developed by Oracle. This system is particularly strong in the area of transactions, data processing, and handling large amounts of data.
PostgreSQL
PostgreSQL, often referred to as Postgres, is an advanced, open-source relational database management system. Initiated in 1986 at the University of California, Berkeley, PostgreSQL has been developed by the open-source community ever since. It’s known for its support of various data types, complex queries, and features like transactions, window queries, and different index types. This makes it a very flexible and powerful tool.
Comparison of the systems
Each of the discussed database management systems has its unique features and benefits. Choosing between them depends on factors like project needs, environment, resources, and team skills.
Conclusion
In the world of data management, there is no “one-size-fits-all” solution. MySQL, DB2, SQL Server, Oracle, and PostgreSQL – each of these systems has its place, and their selection depends on specific requirements and context.
Trends in data management point towards an increasing significance of real-time data, big data sets, and cloud technologies. All these database management systems are evolving to meet these new challenges.
The first, and perhaps most basic, mistake is a lack of awareness or understanding of the GDPR. Despite the regulation having been in effect for several years, some companies still do not fully understand it. They are unsure what these regulations mean for their business. The GDPR is complicated and requires companies to adhere to a number of specific rules regarding personal data. Without a proper understanding of these rules, companies are at risk of non-compliance and potential penalties.
Improper Data Storage and Security
Another common mistake is the improper storage and security of personal data. These data must be stored securely to prevent their loss or theft. In practice, companies need to implement appropriate security measures, such as encryption. They also need to regularly update and test their security systems.
Lack of Effective Data Retention Policy
A lack of an effective data retention policy is another mistake that companies frequently make. The GDPR mandates companies to retain personal data only for the duration necessary for the purposes of its collection. They should not keep the data longer than required. Without a clearly defined data retention policy, companies might keep data either too long or not long enough. Such practices can result in non-compliance.
Failure to Update Data Protection Policies
Data protection law is a dynamic area that is constantly evolving and changing. If companies do not regularly update their policies and procedures, they may easily end up in a situation not compliant with the latest requirements. This means that companies must actively monitor changes in the law and adjust their practices accordingly.
Consequences of Mistakes
Financial Penalties
One of the most severe consequences of non-compliance with the GDPR is financial penalties. If organizations violate GDPR regulations, they can incur fines. These can reach up to 20 million euros or up to 4% of their annual global turnover, depending on which value is higher. For example, in 2019, the UK’s Information Commissioner’s Office imposed a fine of 204 million euros on British Airways for GDPR violations.
Loss of Customer Trust
The loss of customer trust is another important consequence of improper data management. When customers discover that their data wasn’t adequately protected, they might decide against using the company’s services in the future. Such a loss of trust can lead to loss of customers and decrease in sales.
Potential Business Losses
At the end, non-compliance with GDPR and improper data management can lead to huge business losses. Not only do these encompass potential fines and the loss of customers, but they also cover costs tied to rectifying mistakes. This includes implementing new security systems and training staff.
How to Avoid Common Mistakes
Detailed Understanding and Compliance with GDPR Rules
The first step to avoiding these mistakes is understanding and complying with the GDPR rules. Companies should make sure that all individuals handling personal data are familiar with GDPR regulations. They should also ensure these individuals know how to apply these rules. Data protection training should be conducted regularly to ensure that staff are up to date with the latest regulations.
Implementation and Compliance with an Effective Data Retention Policy
Companies should also create an effective data retention policy. This policy should detail how long they should store various types of data. This policy should be regularly reviewed and updated to ensure its compliance with current regulations.
Regular Reviews and Updates of Data Policies and Procedures
Another important step is regular reviews and updates of data-related policies and procedures. This includes not only data retention policy but also data security policy and consent processing procedures. Regular reviews will help ensure that the company’s policies and procedures are up to date with the latest legal requirements.
Training Staff in GDPR Rules and Data Retention
The last but not the least step is ensuring that all staff are adequately trained in GDPR rules and data retention. This will not only help prevent mistakes but also help employees understand why these rules are so important and what could be the consequences of not adhering to them.
Conclusion
Compliance with GDPR and effective data retention are key to maintaining legal compliance and protecting customer trust. By avoiding the most commonly made mistakes, companies can better manage their data, minimize risk, and maximize the benefits of data ownership. It’s a process of continuous learning and adaptation, but the effort is worth it considering the potential consequences.
Welcome to Wizards! We are specialists in data protection, providing effective tools for detecting, anonymizing, and retaining personal data. Our services ensure full compliance with GDPR in terms of personal data protection.
We have 25 years of experience in creating systems for handling sensitive data across various sectors. We utilize cutting-edge technologies such as big data and machine learning. As committed developers and co-owners of the company, we understand the challenges related to GDPR compliance. We want to relieve other specialists from these challenges. This allows them to focus on their own tasks.
f you are looking for tools to manage personal and sensitive data, please contact us. Let’s schedule a conversation. It’s the first step to ensure your company can avoid potential fines. Contact us today.
Understand the Consequences of Violating GDPR
Understand the Consequences of Violating GDPR: Non-compliance with Rules such as Improper Data Retention or Incorrect Anonymization Can Lead to Administrative Sanctions and Financial Penalties.
In accordance with Article 83(5) of the GDPR, violations of the provisions concerning general principles, such as retention or minimization, may result in the imposition of a monetary penalty of up to 20,000,000 EUR or, for enterprises, up to 4% of the total global annual turnover of the previous fiscal year. In the case of both amounts, the higher one applies.
The key aspects of data processing are multifaceted. They include compliance with the law, fairness, and transparency. Purpose limitation and data minimization are also vital. Additionally, accuracy and security are crucial, encompassing aspects such as confidentiality, integrity, and availability. It is also essential to consider the rights, such as deletion or copying of data, and the responsibility for processing.
Check our tools
By choosing Wizards, you protect your company from potentially massive costs arising from GDPR violations. This can be illustrated by the example of British Airways, which was fined 204 million euros for personal data breaches in 2019. Through our collaboration, you can be assured of the safety of your finances, business, employees, and customers.
- Detecto, a system that monitors personal and sensitive data in databases, supporting anonymization and retention processes.
- Revelio, an effective solution for identifying personal and sensitive data in documents and email, promoting the digitization of business processes.
- Nocturno, an advanced system for anonymizing personal data in IT systems, maintaining data consistency between different systems.
- Oblivio, a dedicated system for managing personal data retention within an organization.
With Wizards, you’ll ensure GDPR compliance, protect your company from costly penalties associated with regulation breaches, and safeguard not only your finances but also your business, employees, and customers.
Practical Application of Our Tools
Our tools have real-world applications that translate into the daily functioning of companies. Below, we present specific examples.
Detecto. A telecommunications company was using an advanced ERP solution from one of the leading providers. During an audit, IT had to locate places within the ERP system where personal data were stored. Detecto from Wizards scanned the system’s databases and provided a report with the locations of personal and financial data.
Revelio. In an insurance company, despite using various security tools, there was a leak of customer data and financial information. Revelio helped identify over 1000 files containing more than 100 customer names along with bank account numbers, gathered in the SharePoint repository and on personal computers.
Nocturno. A client was using 20 different IT systems. To handle this complexity, Wizards Nocturno automated the process of creating test and development environments. During this creation, the system ensured that personal and sensitive data were anonymized. Moreover, this anonymization did not affect the qualitative and quantitative parameters of the data. As a result, it allowed for secure testing and software development.
Oblivio. A public institution was processing the data of its employees. Oblivio helped identify which data should be deleted in accordance with data retention regulations. The tool also took care of the anonymization of employee data that needed to be removed from the human resources systems.
Would you like to learn more about the practical applications of our tools? Contact us for a conversation. We have dozens of other examples!
Collaboration Models
Secure your business with one move. Discover our collaboration options and schedule a conversation to safeguard your enterprise.
Collaboration Options:
- Detecto – Identification of personal data in three selected databases for 10,000 PLN.
- Revelio – Identification of personal data in shared resource files (up to 100,000 files) for 10,000 PLN.
- Oblivio – Information on the quantity of personal data subject to retention, stored in your database (applies to 1 process, max 3 systems) for 10,000 PLN.
We offer one-time use options or subscriptions for a period of a month or a year.
With us, it’s worth it!
RODO regulations can be difficult to understand and implement. At Wizards, we focus on accessibility and effectiveness. Our tools, Detecto, Revelio, Nocturno, and Oblivio, tailored to your needs, provide efficient personal data management and full compliance with GDPR.
Don’t let the fear of GDPR hinder your company’s growth. With us, you gain confidence and security, enabling you to focus on what’s most important – your business. So choose Wizards, choose the only proper protection. Then schedule a conversation with us today and safeguard your company from the consequences of GDPR violations.
The problem of processing and retention of personal data in companies is becoming increasingly complicated, especially in the context of GDPR rules. Coming to the rescue is Oblivio – a modern personal data management system.
Personal data management is the process of handling information that can identify a person. Systems like Oblivio not only collect this data, but also process, secure, and delete it in accordance with legal requirements, such as GDPR. This allows individuals, whose data is concerned, to update their information regularly.
System Description
Oblivio is a comprehensive personal data management solution that offers many features that facilitate this process. It also cooperates with Detecto, which helps to quickly find sensitive data. Sensitive data is information of particular importance, such as health data, biometric data, sexual orientation, or religious beliefs. Due to their delicate nature, they require special protection. The processing of this data is strictly regulated and often requires explicit consent from the person whose data it concerns.
Oblivio – Your Partner in Personal Data Management
Oblivio is a tool intended for everyone. From financial companies, through healthcare, to education and trade. Any company that processes any personal data can benefit from our product. This system is the ideal solution for companies that are aware of the challenges associated with processing personal data but do not yet have a developed solution.
Implementation Process
Implementing Oblivio is a process based on three steps: identification of personal data, data analysis, and setting retention rules. The entire process is supervised by our team, which helps to find the best solutions for the company. This saves the company time, while also ensuring that everything is in accordance with the law.
Discover Five Amazing Benefits
Oblivio guarantees benefits. First and foremost, it minimizes the risk associated with GDPR. Additionally, it allows for comprehensive management of personal data. Not only does it provide reports that allow for tracking activities, but it also records all operations, ensuring full accountability. The system also offers automation. Finally, it is worth mentioning that we provide implementation support.
Take That First Step
Our system helps companies avoid not only problems related to data processing but also the risk of GDPR violation. Contact us to learn more.
Processing personal data is an integral part of the digital reality. Companies face many challenges. Cybercrime poses a serious threat. Society is becoming more aware of its privacy. Legal regulations, including GDPR, set specific requirements. Companies need to approach them flexibly and responsibly. However, it’s hard to determine the impact of these elements on the development of artificial intelligence (AI).
Cybercrime
Cybercrime is a significant challenge in today’s world. The intensity of this problem is increasing every year. In business language, terms such as hacking attacks, phishing, and ransomware have become common. Companies processing personal data must invest in IT security systems. They also need to train their teams. This allows them to effectively counter threats. Data security is a priority for every business.
Social awareness of privacy
The right to privacy is a basic human right. In the era of digitization, it becomes even more important. Customers are increasingly aware of their rights. They expect companies to be transparent in data processing. Companies that do not meet these expectations risk losing customers’ trust. This, in turn, can lead to a loss of their loyalty.
The right to privacy is a basic human right. It enables control over information about oneself. It protects against improper use. GDPR, as a European regulation, crucially treats this right.
GDPR obliges companies to protect the processed data, and companies must inform individuals about the purposes of processing their data. They also must allow them access to these data.
GDPR introduces the right to be forgotten, which means that an individual can request the deletion of their data in certain situations. Additionally, the right to data portability is another element of GDPR that allows data to be transferred between service providers.
So, the right to privacy in the context of GDPR is control over personal data. It’s protection against their improper processing.
Increasing legal requirements
GDPR is a key regulation regarding data processing. It introduces many requirements that companies must meet. Severe sanctions threaten violations of these regulations. They can reach up to 20 million euros. Alternatively, it can be 4% of the company’s global turnover. This demonstrates the seriousness of the right to privacy in the European Union. Companies, therefore, need to dedicate resources to ensure GDPR compliance. This is also necessary for other legal provisions.
GDPR and artificial intelligence
Artificial intelligence is an area of progress and innovation. However, it must comply with GDPR regulations. Companies are obligated to inform customers about the processing of their data by algorithms. They also need to provide a “right to explanation” for automatic decisions. Such requirements can slow down the development of AI. At the same time, they can encourage the creation of more “transparent” and ethical AI models.
Solutions and the future
Challenges in data processing are significant, but surmountable. New technologies can increase privacy and data security. Companies investing in advanced AI must integrate GDPR principles at the design stage. This is known as privacy by design. Training employees and building a culture of respect for privacy is key. This applies to all levels of the organization.
Conclusion
Data has become the new “gold” in our world. The ability to manage it is the key to success. GDPR may pose a challenge, but it also offers an opportunity to build trust. Increasing customer loyalty is possible thanks to this. We observe dynamic development of technology, including artificial intelligence. An approach to data that combines innovation and ethics will be the key to future success. What do you think, who will win the battle, GDPR or artificial intelligence?
Data protection is becoming an absolute priority. Anonymization – a process that protects the privacy of individuals whose data we store – is becoming indispensable. In this context, we present Nocturno, a tool that enables efficient data anonymization.
The Leading Tool for Data Anonymization
Nocturno, an advanced anonymization tool, utilizes extensive dictionaries and generators, preserving the characteristics of the database. It anonymizes various systems at once, maintaining data consistency. It handles parallel processing and caching, as well as anonymizing personal data and other sensitive data, such as financial ones.
Who is Nocturno For?
Nocturno is perfect for teams planning to anonymize data. This includes teams creating, developing, and maintaining systems with sensitive data. It is also useful for the software testing process and for transmitting anonymized reports to other systems. Nocturno integrates with our other product – Detecto, which helps to easily find sensitive data.
How Does Nocturno Work?
Nocturno enables rule declaration and built-in generation of common data types. Given the complexity of modern IT systems, this tool is designed to adapt seamlessly to the database structure. Nocturno ensures consistent anonymized data across databases, supporting multiple types and languages.
Key Features of Nocturno
Nocturno is a versatile tool that declares rules for values to be anonymized, an aspect that significantly enhances the data protection process. Moreover, this advanced software is capable of generating the most common data types, demonstrating its adaptability and broad range of functionalities. Nocturno seamlessly adapts to modern IT systems, ensuring robust data handling.
Additionally, one of the vital features of Nocturno is ensuring the consistency of anonymized data across different databases, facilitating a reliable and uniform data protection strategy. As part of its comprehensive functionality, Nocturno supports various types of databases, including MySQL, DB2, SQL Server, Oracle, and PostgreSQL, offering wide-ranging compatibility for diverse IT environments.
Furthermore, Nocturno showcases its capabilities by generating various types of data, such as first and last name, PESEL, NIP, REGON, and address, contributing to its extensive data anonymization solutions. And to cater to the global digital era, it even generates data in different languages, proving its readiness to handle multinational and multicultural data privacy requirements.
Data Anonymization Process
The first step in preparing a company for the implementation of Nocturno is understanding what data is stored and where. An audit of all systems and databases should be carried out to identify which data is sensitive and requires anonymization.
Next, it is worth identifying all the teams that will use Nocturno. All these teams should be informed about the new tool and provided with appropriate training.
It will also be helpful to prepare a test plan. Finally, the company should be ready to introduce changes in its systems.
Benefits of Nocturno
Quality: Importantly, anonymization with Nocturno guarantees data quality, essential for high-quality development and testing processes.
Automation: Nocturno integrates with CI/CD tools like Jenkins for efficient anonymization automation.
Efficiency: Nocturno’s high efficiency ensures a seamless and inconspicuous anonymization process.
Testing Support: Moreover, data anonymization allows software testing without having to operate on actual personal data.
Safety: Nocturno enables effective access restriction to enhance data security.
Convenience: Automatic anonymization allows for effortless database restoration, enhancing everyday convenience.
Integration with Detecto
Nocturno integrates with Detecto, a tool that helps to easily search for sensitive data in various systems and databases.
Data Anonymization with Nocturno
Data anonymization is crucial in today’s digital world. Nocturno enables efficient and safe data anonymization, protecting user privacy and compliance with the law. Although it requires some effort and time, the benefits of data anonymization are significant. In the digital age, where data privacy is key, Nocturno offers a reliable and efficient solution for data anonymization.
It would be hard to miss all the articles on the topic of GDPR and all the various, terrifying sanctions that could be put upon an entity for non-compliance. Few, however, delve into important details, such as the significance of anonymization or data retention, which allow for avoiding all these sanctions and make the work of developers significantly easier. For this reason, we decided to explain in an accessible way what anonymization and retention of personal data are and show why their proper implementation is of such importance in the software development process. Today, let us tackle anonymization.
What is anonymization?
Anonymization is a process that allows you to permanently remove the link between personal data and the person to whom the data relates. Thanks to this, what was previously deemed as personal ceases to be that.
What does it look like in practice?
The definition above becomes less complicated when presented with an example. Let us imagine, for example, Superman – a comic hero from Krypton who wants to hide his identity and blend in with the crowd.
| Name | Superman |
| Occupation | Superhero |
| Origin | Krypton |
During the anonymization process, Superman enters the telephone booth, puts on glasses and a tweed suit, and becomes Clark Kent, a reporter from Kansas.
| Name | Clark Kent |
| Occupation | Reporter |
| Origin | Kansas, USA |
Through the anonymization process, Superman’s data turned into Clark Kent’s, and there is no connection between these two people. This is fictitious data that can be safely used, e.g. in test environments.
The example above illustrates the process of anonymization itself. Let us now consider why it is important that the anonymization is of good quality.
Irreversibility
The foundation of anonymization is its irreversibility. We should never be able to find out what the original data looked like, based on the anonymized data. Clark’s associates should not be able to discover his true identity.
When we anonymize a data set, usually only a fragment of the data will undergo change. However, we must ensure that non-anonymized data does not allow the anonymization process to be reversed for the entire set. In our example, we would not have to change Superman’s favorite color. However, if we do not anonymize his origin, we would certainly cause a sensation.
True to reality
An important qualitative measure of anonymization is also how well it imitates reality. If Superman and all other people in the data set are anonymized as follows:
| Name | X |
| Occupation | Y |
| Origin | Z |
we have no doubt that the process is irreversible, but its usefulness is questionable. Person X does not look like someone who exists in reality, and the nature of the original data has not been preserved. The length of the names were not preserved, and the data itself looks unbelievable with all of the people having the same name. In the case of IT systems, the tester using such data would run into a lot of issues, he would not even be able to distinguish between people.
Repeatability
Another feature of good anonymization is its repeatability. When anonymizing the data set, we want to make sure that each time the data set would be anonymized in the same way. We want Superman to always become Clark Kent, no matter whether it’s the first or the tenth anonymization. This is especially important from the point of view of Quality Assurance. Testers often create test cases based on specific data. If this data were changed every time, the tester’s work would certainly be more difficult!
Integrated systems
Today’s IT world is represented by countless systems connected with each other. Hardly any application can function as a single organism. Systems connect with each other, exchanging data and using each other’s services. Therefore, when approaching anonymization, we must consider the process not only for one system, but for many systems at once. The challenge is for anonymized data to be consistent throughout the entire ecosystem. This means that if the Daily Planet (Clark’s workplace) has a human resources system and a blog, then in both applications Superman will become Clark Kent.
Efficiency
The last key parameter affecting the quality of anonymization, from my point of view, is performance. IT systems process huge data sets measured in gigabytes or even terabytes. Anonymization of such databases can be time consuming, therefore, we must ensure not only security but also good speed of the anonymization process. One of the things Superman learned after arriving on Earth is that time is money. This saying rings even more true in the case of modern IT.
I invite everyone interested in the topic of data retention to read my next article, which I plan on publishing shortly.
Artur Żórawski, Founder & CTO
Good quality tests require good data – data that is the most accurate representation of reality. A copy of production data is very often used for this purpose. Such a dedicated test environment is often used to reproduce tickets, debugging issues with data and performing stress tests. Setting aside the fact that this practice is most often incompatible with the GDPR, while the production environment is monitored and audited like a fortress, and only a few people have access to it, non-production environments are treated much less restrictively. The number of people with access to them (not including the users) is also much larger. Many serious leaks of personal data were not caused by hacking into the “fortress”, but by abuse of these “unprotected settlements”.
In the area of test data, there are usually two extremes – personal data is either processed by testers and developers in production database copies, or, we wait half a year to refresh test environments with artificial data, usually poorly prepared. The solution to this problem could be the implementation of anonymization, but as it turns out, this is not an easy task.
Challenges associated with designing the anonymization process
Simple data masking can work in simple cases, but you can quickly see that this is not enough for applications that we usually work with every day. On the other hand, when reviewing existing solutions, we noticed that they did not meet our needs – most often they did not support mechanisms to maintain data consistency between different databases. It was also difficult to find a solution that supported the automation of the anonymization process. The most popular tools didn’t allow for defining your own generators, not only regarding a single record, but also taking into account the distribution of data. By implementing a solution that meets these requirements yourself, one will quickly encounter obstacles:
- Simple data masking leads to application errors due to potential violation of data formats expected by the application, e.g. personal identification number with a value of 8501XXXXX11 will cause validation errors. This approach also very quickly leads to duplicate data.
- The use of native mechanisms (e.g. Dynamic Data Masking in SQL Server) may be sufficient to view data directly on the database, however, it does not allow collating an anonymized copy of the production environment in a secure manner. In such solutions, the data, although presented as masked, still appears in the database in their raw form.
- A solution based on SQL scripts turns out to be insufficient. Simple, context-free generators (with a number of keys to the order of several hundred thousand and more) will increasingly conflict with previously generated data, causing violations of uniqueness. Solving this problem through lookups will result in a significant degradation of anonymization performance. The introduction of additional database structures to store the generated identities would be cumbersome to maintain. In the case of anonymizing many schemes, including on different database servers, this approach quickly becomes extremely inefficient, and in the case of various database engines, even unusable. Scripts begin to be complex and complicated, requiring more and more time to keep them running.
- Changes in the database structure (e.g. a new version of the application that added or modified tables and columns) will potentially result in the anonymization process being outdated.
Happy medium
However, there exists a happy medium – ensuring free access to high-quality data reflecting the characteristics of production data, while ensuring the security of the solution and compliance with legal regulations. This happy medium is Nocturno – a data anonymization tool that we designed together as a team. While working on this solution, we decided to take care of:
– Maintaining full data consistency – not only within the schema or database, but all data sources within the organization (databases of various suppliers, LDAP, file sources, etc.)
- Reflecting the characteristics of real data
- Supporting automation fully
- Creating the anonymization process easily
- Ensuring full security of the solution – no possibility of reverse anonymization
- Establishing configurability and extensibility – a wide, built-in set of algorithms and generators, with the ability to write custom components
- Enabling versioning of the system’s anonymization process together with the version of the anonymized system (GIT, SVN, etc.)
What do we gain by implementing good-quality anonymization?
By implementing anonymization, we are able to reduce the number of people who have access to personal data to the absolute minimum. Due to the good quality of the anonymized data, its use for software development purposes is transparent and compliant with the GDPR. The process based on Nocturno is easily configurable and maintainable by developers – it can be simultaneously developed in the same codebase as the application.
Nocturno supports two main implementation scenarios:
- Administrative launch of anonymization on the indicated database instance (anonymization on request)
- Automatic process of creating an anonymous copy of the production database – preparation of an anonymous backup ready for use in test environments and by developers
The picture above portrays Nocturno’s role in the automatic process of providing anonymized copies of databases.
More information about Nocturno can be found here: https://wizards.io/en/nocturno-en/. If you have questions about the anonymization process, please feel free to reach out.
Marcin Gorgoń, Senior Software Engineer