Connector Configuration

In some scenarios, it can be useful to provide a system property or an environment variable within a configuration value. For example, you might want to distribute multiple connector instances over different containers and have certain configured parameters adapted according to specific system properties or environment variables, respectively. You can do this by providing a placeholder of the form ${my.system.property} within the value of some configuration option. The connector will resolve this property when reading the configuration, i.e., the placeholder will be replaced by the value of the corresponding system property / environment variable.

However, for security reasons this replacement is disabled by default. A malicious user could misuse this feature to obtain sensitive information about the connector environment such as the host’s operating system, the user under which the connector is running etc. You can mitigate this issue e.g. by restricting the access to the connector UI.

If you want to enable the resolution of system properties and environment variables, set the system property resolvePlaceholdersInPropertyValues to true. This can be done, for example, in the connector’s start script under the bin directory as follows.
In connector.bat (Windows):
set CONNECTOR_OPTS="-DresolvePlaceholdersInPropertyValues=true"
In connector (Linux/Unix):
CONNECTOR_OPTS='"-DresolvePlaceholdersInPropertyValues=true"'

Confluence Configuration

Confluence Source Name Assignment

Configuration Options related to the value of the Source Name.

Name Property Key Description

Source Name Value

raytion.connector.agent.confluence
assign.source.sourceName

The value which the Source Name metadata should be set to.

Confluence Instance Settings

Configuration Options related to establish connection to the target Confluence instance.

Name Property Key Description

URI

raytion.connector.agent.confluence
.instance.uri

URI of the confluence instance to connect to.

Custom Default Headers

raytion.connector.agent.confluence
.instance.customHeaders

Default Headers to be added to HTTP requests to Confluence.

Authentication method

raytion.connector.agent.confluence
.instance.authMethod

Defines how a user authenticates against Confluence. The different methods will be described below. The choices are PERSONAL_ACCESS_TOKEN, BASIC_AUTH, SSL and NONE.

Personal Access Token

The personal access token (PAT) is Confluence’s recommended method of authentication. It is an application-bound, complex password that is automatically rotated periodically by the connector.

There is a maximum number of PATs which may be created per user. It is set to 10 by default, but can be adjusted if needed through the system property atlassian.pats.max.tokens.per.user. Please make sure that this maximum is not reached, if needed by revoking unneeded tokens manually in the Confluence user settings, otherwise the connector will not be able to create new tokens and run into errors.
Name Property Key Description

Initial personal access token (PAT)

raytion.connector.agent.confluence
.instance.patAuthentication.initialToken

You need to manually generate the initial personal access token (PAT) in Confluence and then paste it in here. The initial PAT will be used by the connector to automatically request the required PATs from Confluence according to the settings below. This will require no further manual intervention.

Token name

raytion.connector.agent.confluence
.instance.patAuthentication.tokenName

The connector will request new PATs form the Confluence instance it is connected to using this token name. The generated PATs will show up in Confluence’s configuration with this name.

New tokens are valid for (days)

raytion.connector.agent.confluence
.instance.patAuthentication.tokenTtlInDays

When requesting a PAT the connector will request it to be valid for these number of days.

Renew token before expiry (days)

raytion.connector.agent.confluence
.instance.patAuthentication.gracePeriodInDays

The connector will request a new PAT these many days before the current token expires.

Basic Authentication

Name Property Key Description

Username

raytion.connector.agent.confluence
.instance.basicAuthentication.username

Username to connect to Confluence. This user must be able to log into Confluence and be registered with the Raytion Confluence Plugin.

Password

raytion.connector.agent.confluence
.instance.basicAuthentication.password

Password for the user to connect to Confluence. Encryption is recommended.

SSL Authentication

Configuration options for authenticating the Connector against Confluence via SSL with the help of a KeyStore.

Name Property Key Description

KeyStore File

raytion.connector.agent.confluence
.instance.sslAuthentication.keyStoreFile

A KeyStore to use when connecting to Confluence.

KeyStore Type

raytion.connector.agent.confluence
.instance.sslAuthentication.keyStoreType

The type of the KeyStore-file given. We strongly recommend using PKCS12, as JKS may be deprecated in a future update.

KeyStore Password

raytion.connector.agent.confluence
.instance.sslAuthentication.keyStorePassword

The password to access the keystore.

Key Alias

raytion.connector.agent.confluence
.instance.sslAuthentication.keyAlias

The alias from the KeyStore to use. You can leave this empty if the KeyStore only contains one alias.

Key Password

raytion.connector.agent.confluence
.instance.sslAuthentication.keyPassword

The password to access the Key Alias within the KeyStore.

SSL

SSL Authentication, also known as mutual TLS (mTLS) or double-sided SSL requires the connector to authenticate against Confluence with a certificate that is trusted by the Confluence instance. This is the most secure way of authentication but it is not supported by an out-of-the-box Confluence instance.

Confluence Connection Settings

Configuration Options related to fine-tuning the connection to the Confluence instance.

Name Property Key Description

Socket Timeout

raytion.connector.agent.confluence
.connection.socketTimeout

An inactive connection will be treated as terminated after this duration.

Connection TTL

raytion.connector.agent.confluence
.connection.connectionTimeToLive

Unused connections of the pool will be discarded after this duration.

Requests per second

raytion.connector.agent.confluence
.connection.rateLimit

Maximum amount of requests which are started towards the Confluence instance.

Connection Pool Size

raytion.connector.agent.confluence
.connection.poolSize

Maximum number of parallel requests to Confluence.

Batch Size

raytion.connector.agent.confluence
.connection.batchSize

Number of items that are requested in one request.

Confluence Space Filter Settings

Configuration options determining which Confluence Spaces to crawl.

Name Property Key Description

Included Space Keys

raytion.connector.agent.confluence
.filter.space.includeList

Only the Spaces with keys defined here will be crawled.

If no include filters are defined, all keys are included.

Excluded Space Keys

raytion.connector.agent.confluence
.filter.space.excludeList

Spaces with a key in this list will not be crawled.

This overrides the key being allowed by Included Space Keys.

Included Space Categories

raytion.connector.agent.confluence
.filter.space.includeListCategories

Only the Spaces with at least one category defined here will be crawled.

However, no include filters being defined is synonymous to all categories being included.

Excluded Space Categories

raytion.connector.agent.confluence
.filter.space.excludeListCategories

Spaces with a category in this list will not be crawled.

This overrides the Space being included through Included Space Keys or Included Space Categories.

Confluence Hierarchy Cache Settings

Configuration Options determining how much and long to cache hierarchical data of Spaces and Pages.

Name Property Key Description

Cache Time To Live

raytion.connector.agent.confluence
.cache.maxTtl

Maximum age of cache entries. Larger improves performance and requires more memory.

Maximum Cache Weight

raytion.connector.agent.confluence
.cache.maxWeight

Maximum cache size. Spaces and Pages weigh more the more entries they have. Larger improves performance and requires more memory.

Confluence Type Filter Settings

Configuration Options related to which types of search targets should be generated

Name Property Key Description

Include User Profile Pages

raytion.connector.agent.confluence
.filter.type.extractUserProfiles

User Profile pages will be indexed.

Include Labels Attribute

raytion.connector.agent.confluence
.filter.type.extractLabels

Labels will be indexed.

Extract All Contributors

raytion.connector.agent.confluence
.filter.type.extractAllModifiers

If unchecked, the Contributors attribute of a page or blog post lists only its creator and its last contributor instead of all contributors to it. This requires less calls to Confluence and may speed up indexing.

Comment Handling

raytion.connector.agent.confluence
.filter.type.commentSyncType

NONE: Comments will not be indexed.

INDEX_WITH_PARENT: Comments will be indexed with the parent page or blog post.

INDEX_SEPARATELY: Comments will be indexed separately in an individual document.

Drafts of Pages and Blog Posts

raytion.connector.agent.confluence
.filter.type.extractDraft

Pages and Blog Posts in draft status will be indexed.

Deleted Pages and Blog Posts

raytion.connector.agent.confluence
.filter.type.extractTrashed

Pages and Blog Posts in deleted status will be indexed.

Exclude Attachments by Size

raytion.connector.agent.confluence
.filter.type.maxAttachmentSize

Attachments with a larger size will not have their content indexed.

Attachment File Extension Filter Type

raytion.connector.agent.confluence
.filter.type.attachmentFileExtensionFilterType

Specifies how to filter attachments by their file extension. If it is NONE, no filtering is applied. If the filter type is SPECIFY_INCLUDES, one can specify a list of file extensions that allow only attachments with these file extensions to be included with metadata or metadata and content. If SPECIFY_EXCLUDES is configured, one can specify a list of file extensions that exclude attachments with these file extensions from being indexed with content or content and metadata.

Include Content & Metadata by Attachment File Extension

raytion.connector.agent.confluence
.filter.type.includeContentAndMetadataForFileExtension

Attachments with a file extension from this list will be included entirely.

Include Metadata by Attachment File Extension

raytion.connector.agent.confluence
.filter.type.includeMetadataForFileExtension

Attachments with a file extension from this list will have their metadata without content included.

Exclude Content & Metadata by Attachment File Extension

raytion.connector.agent.confluence
.filter.type.ignoreContentAndMetadataForFileExtension

Attachments with a file extension from this list will be excluded entirely.

Exclude Content by Attachment File Extension

raytion.connector.agent.confluence
.filter.type.ignoreContentForFileExtension

Attachments with a file extension from this list will have their content excluded.

Confluence Label Filter Settings

Configuration Options related to filtering pages and blogs by their labels. Comments and attachments are always included/excluded, if their parent page/blog is included/excluded.

Name Property Key Description

Label Filter Type

raytion.connector.agent.confluence
.filter.label.labelFilterType

Specifies how to filter items by their labels. If it is NONE, no filtering is applied. If the filter type is SPECIFY_INCLUDES, one can specify a list of label names that allow only items tagged with these labels to be included. If the filter type is SPECIFY_EXCLUDES, one can specify a list of label names that exclude items tagged with these labels from being included.

Include Labels

raytion.connector.agent.confluence
.filter.label.includeLabels

Only items tagged with at least one of the labels defined here will be included.

Exclude Labels

raytion.connector.agent.confluence
.filter.label.excludeLabels

Items tagged with at least one of the labels defined here will not be included.

Labels can be configured as follows:

Name Property Key Description

Label Name

raytion.connector.agent.confluence
.filter.label.includeLabels[index].labelName or raytion.connector.agent.confluence
.filter.label.excludeLabels[index].labelName

The name of the label that is included or excluded.

Apply Filter to Children

raytion.connector.agent.confluence
.filter.label.includeLabels[index].applyFilterToChildren or raytion.connector.agent.confluence
.filter.label.excludeLabels[index].applyFilterToChildren

If enabled, the filter also affects the children of the items that pass the filter for this label. If disabled, the filter is only applied to the items themselves. If, for example, a page is tagged with a label that is excluded, the page itself will not be indexed. If the filter for this label is applied to children, the page’s children will also not be indexed. As another example, if a page is tagged with a label that is included, the page itself will be indexed. If the filter for this label is applied to children, the page’s children will also be indexed.

Google Cloud Search Configuration

Account Settings

Configuration Options to specify the service account settings.

Name Property Key Description

Account E-Mail Address for Lookup

raytion.connector.backend
.google-cloud-search
.connection.accountEMail

The E-Mail Address of the user to impersonate. It is used in the principal sync to check if users exist before feeding them to Google Cloud Search. This is not the service account.

Datasource Settings

Configuration Options related to the target Google Cloud Search Datasource. The connector will send any search items to the configured datasource.

Name Property Key Description

Datasource ID

raytion.connector.backend
.google-cloud-search
.datasource.id

The ID of the Google Cloud Search Datasource to index the items to. The IDs can be inspected at admin.google.com.

Default Object Type

raytion.connector.backend
.google-cloud-search
.datasource.objectType

If a document has no meta date objectType set, the value from here is used. The object type specifies which schema object definition, registered for the specified datasource, is used.

Identity Source Settings

Configuration Options related to the target Google Cloud Search Identity Source. The connector will feed the principals to the configured Identity Source.

Name Property Key Description

Identity Source ID

raytion.connector.backend
.google-cloud-search
.identitysource.id

The ID of the Google Cloud Search Identity Source to synchronize the external source system principals. The IDs can be inspected at admin.google.com.

Encoding Settings

Configuration Options to specify encoding settings for indexing items.

Name Property Key Description

Group Encoding Scheme

raytion.connector.backend
.google-cloud-search
.encoding
.groupEncodingScheme

When groups were indexed into the Identity Source using GCDS, their IDs are encoded with Base16 or Base64. When the connector is handling groups it also has to encode group IDs in the correct scheme for the connector to work correctly.

HTTP Settings

Configuration Options related to the HTTP connections to Google Cloud Search.

Name Property Key Description

Max Connections (Content)

raytion.connector.backend
.google-cloud-search
.http.maxConnectionsContent

Maximum number of connections to the Google Cloud Search Content Service.

Max Connections (Security)

raytion.connector.backend
.google-cloud-search
.http.maxConnectionsSecurity

Maximum number of connections to the Google Cloud Search Security Service.

Connect Timeout

raytion.connector.backend
.google-cloud-search
.http.connectTimeout

Timeout to establish a connection.

Read Timeout

raytion.connector.backend
.google-cloud-search
.http.readTimeout

Timeout to read data from an established connection.

Use Proxy

raytion.connector.backend
.google-cloud-search
.connection.useProxy

Flag to connect to Google Cloud Search via a proxy.

Use Proxy Authentication

raytion.connector.backend
.google-cloud-search
.connection.proxy
.useProxyAuthentication

Flag to connect to proxy with authentication.

Proxy URL

raytion.connector.backend
.google-cloud-search
.connection.proxy.baseUrl

URL for targeting Proxy.

Proxy Username

raytion.connector.backend
.google-cloud-search
.connection.proxy.username

Username of the authenticating user against proxy.

Proxy Password

raytion.connector.backend
.google-cloud-search
.connection.proxy.password

Password of the authenticating user against proxy.

Request Settings

Configuration Options related to sending synchronous/asynchronous requests to Google Cloud Search.

Name Property Key Description

Request Mode

raytion.connector.backend
.google-cloud-search
.request.mode

Mode of item requests against Google Cloud Search (Synchronous or Asynchronous)

Request Timeout

raytion.connector.backend
.google-cloud-search
.request.timeout

Timeout of synchronous requests against Google Cloud Search.

Request Period

raytion.connector.backend
.google-cloud-search
.request.period

Time Period in which asynchronous requests against Google Cloud Search should be checked for a response.

Request Workers

raytion.connector.backend
.google-cloud-search
.request.workers

Maximum number of workers requesting the callback for asynchronous requests.

Rate Limit Settings

Configuration Options related to rate limiting to reduce network traffic.

Name Property Key Description

Max Queries Per Second (Content)

raytion.connector.backend
.google-cloud-search
.ratelimit
.maxQueriesPerSecondContent

Maximum number of queries per second to the Google Cloud Search Content Service.

Max Queries Per Second (Security)

raytion.connector.backend
.google-cloud-search
.ratelimit
.maxQueriesPerSecondSecurity

Maximum number of queries per second to the Google Cloud Search Security Service.

Retry Settings

Configuration Options related to the retry mechanism of requests against Google Cloud Search.

Name Property Key Description

Max Retries

raytion.connector.backend
.google-cloud-search
.retry.maxRetries

Maximum number of retries for a request.

Initial Waiting Time

raytion.connector.backend
.google-cloud-search
.retry.initialWaitingTime

Initial Waiting Time for the response of a request.

Delay Factor

raytion.connector.backend
.google-cloud-search
.retry.delayFactor

Factor multiplied to the delay after each failed retry.

Feeding Settings

Configuration Options related to feeding (indexing) items and principals.

Name Property Key Description

Feed only existing GCS Users

raytion.connector.backend
.google-cloud-search
.feeding
.feedOnlyExistingUsers

Flag to indicate if only users who exist in Google Cloud Search should be fed. If a user does not exist and this flag is active then the membership of the user to a group is just not fed. The ACE of an item containing the user is dropped if the user does not exist and this flag is active.

Cache Settings

Name Property Key Description

Maximum Cache Size

raytion.connector.backend
.google-cloud-search
.cache.maxCacheSize

Maximum number of entries in the Cache which stores mappings of principal IDs to GCS resource names.

URL Settings

Name Property Key Description

Additional URL Encoding

raytion.connector.backend
.google-cloud-search
.url.encode

Encode URLs before indexation. This is usually not necessary as the URLs should already be encoded. Default: false.

General Configuration

Database Configuration

Name Property Key Description

Configuration Type

raytion.connector.db.config.type

Supported are PostgreSQL, MS SQL Server, and JDBC URL configuration.

PostgreSQL

Name Property Key Description

Host

raytion.connector.db.config.postgres.host

Domain name or IP address of the database server.

Port

raytion.connector.db.config.postgres.port

Specifies the port number PostgreSQL is listening on, default is 5432.

Database Name

raytion.connector.db.config.postgres.name

Name of the database.

Username

raytion.connector.db.config.postgres.username

Username to authenticate with. The regarding user has to have read and write permissions to the database.

Password

raytion.connector.db.config.postgres.password

Password of the configured database user.

Add Custom Parameter

raytion.connector.db.config.postgres.addParameters

Enables the configuration of additional parameters.

MS SQL Server

Name Property Key Description

Host

raytion.connector.db.config.mssql.host

Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'.

Port

raytion.connector.db.config.mssql.port

Specifies the port number MS SQL Server is listening on, default is 1433.

Database Name

raytion.connector.db.config.mssql.name

Name of the database.

Username

raytion.connector.db.config.mssql.username

Username to authenticate with. The regarding user has to have read and write permissions to the database.

Password

raytion.connector.db.config.mssql.password

Password of the configured database user.

Add Custom Parameter

raytion.connector.db.config.mssql.addParameters

Enables the configuration of additional parameters.

JDBC URL

Name Property Key Description

URL

raytion.connector.db.config.jdbc.url

JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: jdbc:postgresql:<host>:<port>/<database>

Username

raytion.connector.db.config.jdbc.username

Database Username to read and write to database.

Password

raytion.connector.db.config.jdbc.password

Database Password for the specified user

Traversal Configuration

Name Property Key Description

Traversal History Length

raytion.connector.agent.traversal
.store.historyLength

Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100)

Include Checksum

raytion.connector.agent.traversal
.pipeline.includePipelineChecksum

If enabled, any changes made to the pipeline e.g. configuration, the subsequent incremental run triggers a refeed of all items.

Change Processing Interval

raytion.connector.agent.traversal
.changeprocessing.interval

Interval between change processing traversals.

Resume on Start

raytion.connector.agent.traversal
.resume.resumeTraversalOnRestart

If enabled, any traversals in paused state are automatically resumed after the connector restart. Otherwise, the traversal remains in paused state.

Number of Traversal Workers

raytion.connector.agent.traversal
.workers.worker

Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10)

Traversal Job Poll Interval

raytion.connector.agent.traversal
.workers.jobPollInterval

Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms)

Completion Timeout

raytion.connector.agent.traversal
.workers.completionTimeout

If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m)

Executor Size

raytion.connector.agent.traversal
.execution.executorSize

The executor size restricts the max. number of concurrent running traversals.

Queue Size

raytion.connector.agent.traversal
.execution.queueSize

The queue size restricts the max. number of queued traversals. If the value is exceeded, the connector rejects further traversal requests until the queue size is below the configured size.

Traversal Jobs

Name Property Key Description

Job Timeout Check Frequency

raytion.connector.job-broker.heartbeatPeriod

Configures how often the connector checks for timed out jobs.

Job Timeout

raytion.connector.job-broker.heartbeatTimeout

The duration for which a job can stay idle before it is timed out.

Job Cache Size

raytion.connector.job-broker.jobPollCacheSize

Max. cache size of Jobs waiting for processing in memory. When cache is empty, next batch is fetched.

Security Configuration

Request Restriction Settings

Name Property Key Description

Accepted Host Names

raytion.connector.security
.requests.acceptedDomains

A list of domains (+ port) that are allowed as host names in the headers of HTTP requests to the connector. This means that you can access the connector only via a URL that employs one of the configured domains. Each entry must have the format domain:port. Examples:

  • localhost:16160

  • connector.company.com:16160

If no domains are configured (the default), then you can use any domain via which the connector host is reachable.

Principal Aliaser Configuration

Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.

Custom Aliaser Disabled

If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.

Custom Aliaser Enabled

If custom aliasing is enable then there are four types of aliaser avaialble:

Simple XML Table Aliaser

Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.

Name Description

XML Mapping File

Browse and upload or drag and drop.

Sample XML mapping file:

<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
    <entry keyValue="user1">user1@raytion.com</entry>
    <entry keyValue="user2">user2@raytion.com</entry>
    <entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser

Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.

Name Property Key Description

Pattern

raytion.connector.aliaser.aliasers[*]
.replacer.pattern

The regular expression to match, this is the part that will be replaced. If braces (…​) are used in the pattern then the matched value can be retrieved using $1

Substitute String

raytion.connector.aliaser.aliasers[*]
.replacer.substituteString

String to replace the matching part of the find string. Matched value is accessed by employing $1
Regex Extractor Aliaser

Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.

Name PropertyKey Description

Pattern

raytion.connector.aliaser.aliasers[*]
.extractor.pattern

The regular expression to match, this is the part that will be inserted into the new value. If braces (…​) are used in the pattern then the matched value can be retrieved using $$

Insert-Into String

raytion.connector.aliaser.aliasers[*]
.extractor.insertIntoString

String to replace the matching part of the pattern. Matched value is accessed by employing $$
LDAP Aliaser

Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.

Name Property Key Description

Host

raytion.connector.aliaser.aliasers[*]
.ldap.host

Fully Qualified Domain Name of an LDAP server

Port

raytion.connector.aliaser.aliasers[*]
.ldap.port

Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL

AccountDN

raytion.connector.aliaser.aliasers[*]
.ldap.bindAccountDN

AccountDN for bind to LDAP

Password

raytion.connector.aliaser.aliasers[*]
.ldap.password

Password part of credentials

Input Field

raytion.connector.aliaser.aliasers[*]
.ldap.inputField

The Active Directory attribute name for this equality filter

Search Root DN

raytion.connector.aliaser.aliasers[*]
.ldap.baseDN

Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree

Output Field

raytion.connector.aliaser.aliasers[*]
.ldap.outputField

Attribute that should be returned in result entries