博客

在多数分布MariaDB的设施默认设置使用保守的缓冲区的大小和内存使用。

使用保守的缓冲区大小和内存使用率

Cloudera的数据库管理服务器，监控活动，报告管理，Cloudera 导航，Hive 的元数据

二进制日志提供的好处例如：在MariaDB 恢复后的复制或者点对点回复

这配置文件是Cloudera 推荐的设置

1.
如果 MariaDB 服务正在启动，先停止它。

$ sudo service mariadb stop

2.
将旧的 InnoDB 日志被分到 /var/lib/mysql/ 以外的路径

日志文件地址： /var/lib/mysql/ib_logfile0

/var/lib/mysql/ib_logfile1

3.
确定 marigDB 配置文件 my.cnf 的地址：

4.
修改 my.cnf 使他满足下列要求:

o
为了防止死锁，设置隔离级别为read committed。

o
在多数分布式系统中MariaDB 默认使用保守的缓冲区大小和内存使用率。 Cloudera Management Service roles need high write throughput because they might insert many records in the database. Cloudera recommends that you set theinnodb_flush_method property to O_DIRECT.

o
Set the max_connections property according to the size of your cluster:


Small clusters (fewer than 50 hosts) – You can store more than one database (for example, both the Activity Monitor and Service Monitor) on the same host. If you do this, you should:


Put each database on its own storage volume.


Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases, set the maximum connections to 250. If you store five databases on one host (the databases for Cloudera Manager Server, Activity Monitor, Reports Manager, Cloudera Navigator, and Hive metastore), set the maximum connections to 550.


Large clusters (more than 50 hosts) – Do not store more than one database on the same host. Use a separate host for each database/host pair. The hosts need not be reserved exclusively for databases, but each database should be on a separate host.

o
Binary logging is not a requirement for Cloudera Manager installations. Binary logging provides benefits such as MariaDB replication or point-in-time incremental recovery after database restore. Examples of this configuration follow. For more information, see The Binary Log.

Here is an option file with Cloudera recommended settings:

根据簇的大小设置合适的max_connections值

允许每个数据库最大连接数为100个，然后添加50个额外的连接

5. If AppArmor is running on the host where MariaDB is installed, you might need to configure AppArmor to allow MariaDB to write to the binary.

6. Ensure the MariaDB server starts at boot.

如果MariaDB所在主机上运行有AppArmor，你可能需要配置AppArmor允许MariaDB写入二进制

Start the MariaDB server:

启动 MariaDB 服务

chown -R mysql.mysql

$ sudo /usr/bin/mysql_secure_installation

[…]

Enter current password for root (enter for none):

OK, successfully used password, moving on…

[…]

Set root password? [Y/n] y

New password:

Re-enter new password:

Remove anonymous users? [Y/n] Y

[…]

Disallow root login remotely? [Y/n] N

[…]

Remove test database and access to it [Y/n] Y

[…]

Reload privilege tables now? [Y/n] Y

All done!

在 Cloudera Manager Server 主机上面安装 JDBC 驱动，也需要在 活动监控， 报告管理， Hive元数据服务，烧饼服务， cloudera 导航审查de 每一个主机角色上面安装。

Creating Databases for Activity Monitor, Reports Manager, Hive Metastore Server, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server

为Cloudera Manager 各角色创建数据库

为需要数据库的组件创建用户组账户

create database clouderaManager default character set utf8;

drop database clouderaManager;

grant all on clouderaManager.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

create database cm_amon default character set utf8;

create database cm_rman default character set utf8;

create database cm_metastore default character set utf8;

create database cm_sentry default character set utf8;

create database cm_nav default character set utf8;

create database cm_navms default character set utf8;

— =========================================================

grant all on cm_amon.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_rman.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_metastore.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_sentry.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_nav.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_navms.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

grant all on cm_amon.*, cm_rman.*, cm_metastore.*, cm_sentry.*, cm_nav.*, cm_navms.* TO ‘cmroot’@’%’ IDENTIFIED BY ‘admin’;

Role
Database
User
Password

Activity Monitor
amon
amon
amon_password

Reports Manager
rman
rman
rman_password

Hive Metastore Server
metastore
hive
hive_password

Sentry Server
sentry
sentry
sentry_password

Cloudera Navigator Audit Server
nav
nav
nav_password

Cloudera Navigator Metadata Server
navms
navms
navms_password

=================================

[mysqld]

transaction-isolation = READ-COMMITTED

# Disabling symbolic-links is recommended to prevent assorted security risks;

# to do so, uncomment this line:

# symbolic-links = 0

key_buffer_size = 32M

max_allowed_packet = 32M

thread_stack = 256K

thread_cache_size = 64

query_cache_limit = 8M

query_cache_size = 64M

query_cache_type = 1

max_connections = 1000

#expire_logs_days = 10

#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace ‘/var/lib/mysql/mysql_binary_log’ with an appropriate path for your system

#and chown the specified folder to the mysql user.

log_bin=/data/mysql_dir/mysql_binary_log

binlog_format = mixed

read_buffer_size = 2M

read_rnd_buffer_size = 16M

sort_buffer_size = 8M

join_buffer_size = 8M

# InnoDB settings

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit  = 2

innodb_log_buffer_size = 64M

innodb_buffer_pool_size = 4G

innodb_thread_concurrency = 8

innodb_flush_method = O_DIRECT

innodb_log_file_size = 512M

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

===========

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_c.html

Create Users

The Cloudera Manager Server and managed services require a user account to complete tasks. When installing Cloudera Manager from tarballs, you must create this user account on all hosts manually. Because Cloudera Manager Server and managed services are configured to use the user account cloudera-scm by default, creating a user with this name is the simplest approach. This created user, is used automatically after installation is complete.

To create user cloudera-scm, use a command such as the following:

Cloudera Manager Server 和管理服务需要一个用户账户来完成任务。

当通过 tar 包 安装Cloudera Manager时，你必须在所有机器上手动创建这个用户账户。

因为 Cloudear Manager Server 和 管理服务是通过默认账户 cloudear-scm 来配置的。

用这个名字创建用户是最简单的途径。在安装完成之后,这个创建的用户是自动使用的.

—————————————————————————————————–

$ sudo useradd –system –home=/opt/cloudera-manager/cm-5.5.3/run/cloudera-scm-server –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm

—————————————————————————————————–

Configure a Database for the Cloudera Manager Server

为 Cloudera Manager Server 配置一个数据库

Depending on whether you are using an external database, or the embedded PostgreSQL database, do one of the following:

这取决于你使用的是外部数据库，还是内嵌的 PostgreSQL 数据库， 请执行以下操作之一：

External database – Prepare the Cloudera Manager Server database as described in Preparing a Cloudera Manager Server External Database.

外部数据库 – 准备 Cloudera Manager Server 数据库

Embedded database – Install an embedded PostgreSQL database as described in Installing and Starting the Cloudera Manager Server Embedded Database.

内嵌式数据库 – 安装一个内嵌 PostgreSQL 数据库.

[mariadb]

name=MariaDB

baseurl=http://yum.mariadb.org/10.1.12/centos6-amd64/

gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck=1