# Oncrawl[¶](https://www.dataiku.com/product/plugins/oncrawl/)

## Oncrawl plugin for Dataiku provides a recipe that allows you to easily export URLs or aggregated data from crawls and log monitoring events[¶](https://www.dataiku.com/product/plugins/oncrawl/)

## Plugin information[¶](https://www.dataiku.com/product/plugins/oncrawl/)

|  |  |

| --- | --- |

| Version | 1.0.0 |

| Author | Oncrawl |

| Released | 2021-06 |

| Last updated | 2021-06 |

| License | Apache Software License |

| Source code | Github |

| Reporting issues | Github |

## Description[¶](https://www.dataiku.com/product/plugins/oncrawl/)

Oncrawl plugin for Dataiku provides a recipe that allows you to easily export URLs or aggregated data from crawls and log monitoring events.

Export data from Oncrawl platform to bring data science into your SEO management: use machine learning to detect anomalies when crawling your website, or predict the evolution of your traffic or the behavior of the  search engine crawlers… and so many innovative projects to improve your SEO.

## How to use[¶](https://www.dataiku.com/product/plugins/oncrawl/)

### Prerequisites[¶](https://www.dataiku.com/product/plugins/oncrawl/)

* API access in your subscription.

* Live crawls or logs.

### Create a new dataset using the plugin.[¶](https://www.dataiku.com/product/plugins/oncrawl/)

* Choose project source:  all or only a specific one

* Choose data kind among pages, links or logs

* Define a crawl or logs timeframe

* Choose a crawl config.

An empty crawl configs list means that you have no crawl available for the selected project or timeframe, adjust these parameters

* Choose crawl source: all, only a specific one, or the last one into the date range selected

* Aggregations: edit JSON object to define your own aggregations: write one or several OQL queries into the array of aggregate queries. OQL language for aggregations is described here

* URL export: edit JSON object writing your own OQL query to filter the output URL list. OQL language for URL export is described here.  Optionally you can set a specific list of fields you want to export instead of exporting all available fields. It means that if you are only interested by titles for example, your query looks like this:

|  |

| --- |

| {
“oql”: {
“field”: [
“fetched”,
“equals”,
“true”
]
},
“fields”: [
“title”,
“title\_evaluation”
]
} |

A list of fields is available here.  The recipe stores the columns “title” and “title\_evaluation” in the output dataset:

Output dataset only with the fields title and title\_evaluation

#### Install In DSS[¶](https://www.dataiku.com/product/plugins/oncrawl/)

To install the plugin, open the  Apps menu, click Plugins and search for OnCrawl.

Alternatively, you can download a zipped version here.

### Get the Dataiku Data Sheet[¶](https://www.dataiku.com/product/plugins/oncrawl/)

Learn everything you ever wanted to know about Dataiku (but were afraid to ask), including detailed specifications on features and integrations.
