Map an information system

1) oA-Data-lineage-system: data lineage in databases

Starting from any "data point" (a field of a table, a table, a file, a schema), {openAudit} makes it possible to understand its origin and its uses, on-premise and in the Cloud, through technical, multi-technology data lineage. The underlying analyzes are automatically replayed daily.

The information relating to the scheduling of the chains is available, and the uses of each "data point" are defined on hover: who consults what information, when, via which tools, what is of major interest in a compliance framework.

{openAudit} offers different graphical representation modes for its data lineage (business / IT)

Use cases:
Share a detailed understanding of flow construction, identify breaks and correct them, Data Loss Prevention (DLP), BCBS 239 (Basel III), GDPR, etc.

find out more

Data lineage in databases

A real data lineage in underlying layers, exhaustive and totally automated, which presents multiple views according to needs.

Resolving data lineage breaks

> Views: if they are stored, {openAudit} will read them, even if they are stacked (views of views of views...).

> Dynamic SQL: If {openAudit} fails to resolve it directly, the dynamic SQL is resolved with runtime parameters, or with runtime logs.

> Other: in the event of transfer of information by FTP, or when the database schema is not specified (this is the case in many ELT/ETLs), {openAudit} resolves these breaks by structural recognition , where {openAudit} reads the Batch / Shell.

Dynamicly combine different data transformation technologies

> {openAudit} analyzes all processing technologies (object language/procedural, ELT/ETL), on-premise or Cloud, and combines them in a single data flow, at the level the finer. The drill through provides access to the code.

> The process is dynamic, operated in delta mode, daily, and therefore synchronized with the information system.

Different levels of analysis

> Cloud of points: this view allows to instantly know the uses of a data point by disregarding transformations. It is also possible from a use (a dashboard, data from a dashboard, a query), to instantly identify its operational sources.

> Mapping: this view allows from any data point (field, table) to display a complete mapping of the upstream or downstream flow, i.e. from the operational sources to the exposure of the data (dataviz, query...). The information used is highlighted, and the uses of the information are specified on the flyover (who consults the data, when, how).

> Granular data lineage: this view makes it possible to gradually follow the deployment of data in the information system from a data point by iterative clicks, or conversely to go back to operational sources. Each transformation (ELT/ETL job, procedural code/object) can be analyzed with the “drill through”. The precise details of the uses of the data (who consults it, when, how, etc.) are specified.

Map an information system

2) oA-Data-lineage-viz: data lineage in the reporting layer

{openAudit} makes it possible to understand all of the company's data visualization technologies on a single impact analysis interface: it is a grid that makes it possible to understand the staging between the each of the constituent elements of the dashboard and the physical source field (or the view): from the dashboard cell, to the query that query the database, via the semantic layer if there is one, etc.
Data lineage in the dashboard or in the underlying layers can be triggered from this interface.

Thus, all the internal management rules for data visualization technologies are highlighted and shared with everyone. This data lineage in the data visualization layer can be attached to that of the underlying flows.

Use cases: Shedding light on complex management rules, analyzing the impact between a physical field and dashboard data, etc.

find out more

Data lineage in the reporting layer

Some dataviz technologies use semantic layers to create intelligibility for the business, and thus give it autonomy. These semantic layers create abstraction: the underlying physical fields are difficult to identify, which makes sourcing complex.

Furthermore, dataviz technologies often query views, views of views… which again complicates sourcing.

As dataviz technologies multiply, real multi-technology impact analyzes (or sourcing) are complex to operate.

In addition, data visualization technologies now make it possible to do data preparation in large proportions, which creates significant opacity.

Technical answers

> {openAudit} operates a data lineage in the dataviz layer, in the expressions, in the variables, etc., to identify the fields directly or indirectly in source of a data point.

> {openAudit} analyzes the content of views to identify the physical fields that are the source of data for the dataviz layer, even if the views are stacked.

> {openAudit} combines the analyzes of the different dataviz technologies in the same grid, which will allow business and IT to carry out impact analyzes between all the underlying layers and all the data vizualisation tools. Merely.

Optimize a system

1) oA-Optimization-system: detect "dead branches" in feeding chains

Thanks to an analysis of the uses of information, associated with the flow of information (data lineage), {openAudit} identifies useless flows and the associated "data points" (tables / files).

On average, 50% of what is stored in the information system has no added value. These are countless "dead branches" composed of code, tables, views, files, which are wrongly maintained in the systems with considerable impacts: inertia of legacy systems, maintenance costs. And for Cloud systems, unsustainable bills with strong environmental impacts.

Use cases:
Decommissioning of dead branches before migration, rationalization of a system to reduce maintenance, FinOps, GreenOps for a more virtuous IT.

find out more

Detect "dead branches" in feeding chains

A large part of the content of information systems has no added value (replicated, obsolete), with significant impacts: maintenance, licenses, technical migrations made impossible, costs, etc.

Technical answers

> {openAudit} analyzes audit database logs and the data vizualisation layer to find out what data is actually being used.

> From fields used in databases to feed data visualization tools, or ad hoc queries (ODBC, JDBC), or from specific ETL/ELT flows, {openAudit} identifies data flows that are in source, i.e. the "living branches" of the information system. In contrast, {openAudit} identifies "dead branches", i.e. tables, procedures, ETL/ELT jobs that build information without ever being used.

> {openAudit} implements these analyzes in a dynamic way, and thus allows, by creating a substantial depth of history, to formally identify the branches which are continuously unused, with all that they concentrate: tables, files, procedures, ELT/ETL jobs. Mass decommissioning can take place in record time.

> In the Cloud, through an analysis of certain logs, {openAudit} identifies the cost of keeping dead branches in the system. The saveable machine resource is also highlighted. The company can enter into a FinOps and GreenOps logic.

Optimize a system

2) oA-Optimization-viz: detect useless, replicated dashboards

The data visualization layer also contains unnecessary complexity. Users naturally tend to copy and paste adding minor nuances.

{openAudit} will analyze the sources, the queries, the expressions, the variables, the data finally displayed, to allow optimizations. Obsolete, broken and replicated dashboards are detected and can be archived or deleted.

We have developed features for SAP BO that allow to perform certain bulk actions in an automated way to find an essentialized platform almost instantly.

Use Cases:
Massive rationalization of the data visualization layer to reduce costs, reduce the risk of errors, improve the intelligibility of the data visualization layer.

find out more

Detect useless, replicated dashboards

The data visualization layer is often complex, because we stack technologies, but also we modify management rules, we replicate, we overload formulas, etc. At the very end, it is the very quality of the indicators that suffers, the very objectivity of a dashboard.

Technical answers

> {openAudit} will directly parse the files of the data visualization solution to retrieve the intelligence, the structure of the dashboards and the semantic layer if there is one;

> {openAudit} will also access the repository to keep IDs consistent between the different dashboard objects (semantic layer, query, dashboard, others);

> An {openAudit} probe will retrieve certain logs from audit databases that are associated with data visualization solutions.

From there, {openAudit} will allow:

> To compare dashboards with each other and detect replication on different criteria;

> To detect the obsolescence of dashboards;

> To identify broken formulas;

> To detect unnecessary queries;

This in addition to the impact analysis grids, and the data lineage.

Use case #3 -
Migrate to the Cloud

1) oA-Migration-system: migrate procedural code

Technological migrations of object/procedural languages are often so complex that companies prefer to stack technologies rather than decommission them.
However, they struggle to maintain these languages due to a lack of experts capable of reverse engineering.
Nowadays, the craze for the Cloud is changing this paradigm, and more and more companies are looking to get rid of these legacy languages quickly, with no other solution than to start hazardous and costly migrations.

Use cases:
DBMS change, Cloud migration, maintainability of a legacy system, etc.

find out more

Migrate procedural code

Large companies have always accumulated processing technologies. There is a continuous piling up, because the removal of a technology often presents too many risks. But the skills associated with it are becoming scarce, and retro-documentation is rarely in place.
At some point, companies have to get started! These can be consultancy projects, long, expensive, and risky. We think it's better to automate the process.

Technical answers

> {openAudit} will "parse" the source code, it will break down all the complexity of the code using a grammar allowing exhaustive and ultra-granular analyses. All subtleties will be taken into consideration,

> {openAudit} deduces the overall kinematics and intelligence, which will be reconstructed in an algorithmic, agnostic tree. On this basis, {openAudit} will produce "standard SQL",

> Then the intelligence will be reconstructed at least in the specific SQL of the target database (e.g. BigQuery for Google, Redshift for Amazon, Azure SQL for Microsoft, etc.),

> All complex processing that cannot be reproduced in simple SQL will be driven by a NodeJS executable. Typically "For Loop" cursors, variables, "If Else" conditional code, "Switches", procedure calls, etc.,

> {openAudit} produces "Yaml" files (intuitive files). Thus, the understanding of complexity is shared with as many people as possible.

> Optionally, new orchestration mechanisms can be implemented, to deconstruct the sliders of sliders (the loops of loops) to optimize the transformation chains.

Use case #2 -
Migrate to the Cloud

2) oA-Migration-viz: migrate dashboards to the Cloud

How to decommission an outdated dataviz technology because it is too static, expensive, incompatible with the target architecture, especially in the Cloud? How can we go serenely towards the tools of tomorrow, also acclaimed by business lines?

{openAudit} allows almost automated migrations between different dataviz technologies, to save infinite time and avoid damaging regressions, and quite simply to enable these projects!

Use cases:
Migrate SAP BO to Looker, or PowerBI, migrate Qlik Sense to Power BI, etc., many scenarios are possible!

find out more

Migrate dashboards to the Cloud

Most dataviz tools have two things in common: a semantic layer that interfaces between IT and the business, and a dashboard editor.
We rely on the automated reverse engineering of {openAudit} to deconstruct the complexity in source, allowing us to re-address it in the target technology.

Methodology

> {openAudit} will be able to feed the target technology from the single semantic layer, a kind of pivot model. This model will have been generated automatically from the dataviz tools to be decommissioned,

> The structure of the initial dashboard will also have been analyzed by {openAudit}, and it can also be transcribed into the target technology,

> Thus sprawling migration projects, difficult, or impossible to implement can be implemented in record time.

{openAudit}

Laurent Ostiz

Dominique Lhermitte

Christophe Jacri

Data catalogue partner:

3 use cases, 6 features, to transform a system

Use case #1: Map a system

Use case #2: Optimize a system

Use case #3: Migrate to the Cloud

Map an information system

1) oA-Data-lineage-system: data lineage in databases

Data lineage in databases

Map an information system

2) oA-Data-lineage-viz: data lineage in the reporting layer

Data lineage in the reporting layer

Optimize a system

1) oA-Optimization-system: detect "dead branches" in feeding chains

Detect "dead branches" in feeding chains

Optimize a system

2) oA-Optimization-viz: detect useless, replicated dashboards

Detect useless, replicated dashboards

Use case #3 - Migrate to the Cloud

1) oA-Migration-system: migrate procedural code

Migrate procedural code

Technical answers

Use case #2 - Migrate to the Cloud

2) oA-Migration-viz: migrate dashboards to the Cloud

Migrate dashboards to the Cloud

Methodology

discover some features

News

2022-04-07Ellipsys was at the FIC 2023 !

2022-12-01{openAudit} is part of the UGAP catalog!

2022-09-27Conference: transforming your IS thanks to Data Lineage!

Data inventory:

Data inventory:

Probes and log parsing:

Probes and log parsing:

Introspection of the dataviz layer:

Introspection of the dataviz layer:

Reverse engineering on the code:

Reverse engineering on the code:

{openAudit} ?

specificities:

Scheduler parsing:

Scheduler parsing:

Contact

3 use cases, 6 features,
to transform a system

Use case #1:

Map a system

Use case #2:

Optimize a system

Use case #3:

Migrate to the Cloud

Use case #3 -
Migrate to the Cloud

Use case #2 -
Migrate to the Cloud

2022-04-07
Ellipsys was at the FIC 2023 !

2022-12-01
{openAudit} is part of the UGAP catalog!

2022-09-27
Conference: transforming your IS thanks to Data Lineage!

{openAudit}
?