markdown:
Automated Instance failover Using IBM DB2 High Availability Instance Configuration Utility (db2haicu) on shared storage (AIX/Linux) Date: March 18, 2010 Version: 2.0 Authors: Abhishek Iyer (
[[email protected]]) Neeraj Sharma ( [[email protected]]) Abstract: This is a step by step guide for setting up an end to end HA DB2 Instance on shared storage with TSA using the db2haicu utility for Linux. Same procedure is applicable to AIX as well when used with equivalent commands for AIX. Table of Contents 1. Introduction and Overview ............................................................................................. 4 2. Before we begin .............................................................................................................. 4 2.1 Hardware Configuration used ................................................................................... 4 2.2 Software Configuration used .................................................................................... 4 2.3 Overall Architecture.................................................................................................. 5 2.3.1 Hardware Topology ........................................................................................... 5 2.3.2 Network Topology ............................................................................................. 6 3. Pre Configuration steps................................................................................................... 7 3.1 Configuring the /etc/hosts file:.................................................................................. 7 3.2 Configuring the db2nodes.cfg file: ........................................................................... 8 3.3 Configuring the /etc/services file .............................................................................. 8 4. Configuring the Standby Node ....................................................................................... 9 4.1 NFS Server settings (configuring /etc/exports file) .................................................. 9 4.2 NFS Client settings (updating /etc/fstab file)............................................................ 9 4.3 Storage failover settings (updating /etc/fstab file) .................................................. 10 5. Configuring a DB2 Instance for HA using db2haicu.................................................... 13 5.1 Procedure of running db2haicu............................................................................... 13 5.2 Appendix for db2haicu ........................................................................................... 29 6. Configuring the NFS Server for HA............................................................................. 30 7. Post Configuration steps (at the customer site)............................................................. 34 8. Cluster monitoring ........................................................................................................ 37 9. Lessons learnt during HA implementations.................................................................. 40 9.1 Hostname conflict ................................................................................................... 40 9.2 Prevent auto-mount of shared file systems ............................................................. 40 9.3 Preventing file system consistency checks at boot time ......................................... 40 1. Introduction and Overview This document is aimed to serve as an end to end guide for configuring a database instance as a HA (high available) instance across shared storage. A high available database instance across shared storage is typically needed in a Balance Warehouse (BCU) environment where one standby node acts a failover node for all the data and admin nodes. This will be discussed more in the Overall Architecture section below. The implementation discussed in this document is based on the DB2 High Availability (HA) feature and the DB2 High Availability Instance Configuration Utility (db2haicu) which is available in DB2 Version 9.5 and beyond. This utility uses the Tivoli System Automation (TSA) cluster manager to configure the shared database instance. A user can use this utility in the following two modes: • Interactive Mode: The user would need to provide all the required inputs step by step as prompted on the command line by the utility. • XML Mode: In this mode all the inputs need to be written into an XML file which the utility would parse to extract the required data. This document explains how to configure a shared instance on shared storage using the step by step interactive mode (in section 5 below). 2. Before we begin It is important that you go through the setup information before moving on to the actual HA configuration steps. The hardware used in the current implementation is a D5100 Balanced Warehouse (BCU) which has the following nodes (Linux boxes): • • • • 1 Admin node (admin01) 3 Data nodes (data01,data02 and data03) 1 Standby node (stdby01) 1 Management node (mgmt01) 2.1 Hardware Configuration used • • • Each node is an x3650 server with Quad-Core Intel Xeon Proc X5470 (3.33 GHz). All nodes have 32 GB memory except the management node which has 8 GB. The Admin and the Data nodes have 24 external hard disks each of 146 GB capacity. 2.2 Software Configuration used • • • DB2 Linux Enterprise Server Edition DWE IBM Tivoli System Automation 9.7.0.1 9.7.0.0 3.1.0.0 Operating System SUSE Linux Enterprise Server VERSION = 10 PATCHLEVEL = 2 Kernel Information 2.6.16.60-0.21-smp 2.3 Overall Architecture This section describes the overall architecture in terms of hardware, network topology of the highly available database cluster under implementation. 2.3.1 Hardware Topology In a typical D5100 Balanced Warehouse environment, the standby node is designed to be the failover node for the admin node as well as the data nodes. The management node is not a part of the HA cluster, as it is only used to manage the other nodes using the cluster system management utilities. Hence we’ll not be referring to the management node henceforth in this document. As mentioned in the hardware configuration above, each of the admin and data nodes have their respective storage disks which are connected through Fiber Optic cables (shown by red lines in Figure 1 below). The Standby node would be configured to take control of the storage mount points of the failed node in the event of a failover (shown by red dotted lines in Figure 1 below). Even though the database instance resides across all of the admin and data nodes in a balanced warehouse, any external application would typically connect only to admin node which internally acts as the co-coordinator node. An NFS server runs on the admin node and all other data nodes are NFS clients. The communication between the admin and data nodes takes place using the Gigabit Ethernet network (shown in purple lines in Figure 1 below). In the event that a data node fails over to the standby node, the standby node must start functioning as an NFS client and in case the admin node fails over, the standby node must function as the NFS server and take the role of the coordinator node. The step by step configuration of the standby node for each of these failover scenarios is described in detail in the following sections. Figure 1: Hardware Topology 2.3.2 Network Topology A D5100 Balanced Warehouse typically has the following networks: • Cluster management network. This network supports the management, monitoring and administration of the cluster. The management node (mgmt01) uses this network to manage the network using the Cluster System Management Utilities. This network may or may not be made highly available. In the current implementation this network is on subnet 192.168.111.0 (shown in brown lines in Figure 1 above) and we would be making it highly available. • Baseboard management controller. Additionally there is a service processor network that is linked to this network. This service processor, called the baseboard management controller (BMC), provides alerting, monitoring, and control of the servers. This is port 1 of integrated network ports. This network is port is common to Cluster management network. • DB2 fast communications manager (FCM) network. The DB2 FCM network is the network which is used for internal database communication between database partitions on different physical servers. This Gigabit Ethernet network supports FCM traffic as well as the NFS instance directory used in a DB2 with Database Partitioning Feature (DPF) configuration. This network is made highly available as all the data transfers between different nodes happen over this network. In the current implementation, this network exists on subnet 172.16.10.0 (shown in purple lines in Figure 1 above). • Corporate network (optional). This network allows external applications and clients to access and query the database. Typically, external applications would only require connecting to the admin node which would internally coordinate with all the other data nodes on the FCM network, but in some cases for more flexibility, data nodes are also made reachable on the corporate network. In the current implementation, only the admin and standby nodes are reachable on the corporate network on the subnet 192.168.253.0 (shown with green lines in Figure 1 above). Standby is made available on the corporate network to provide an alternate route for the external applications in case the admin node goes down. 3. Pre Configuration steps There are some pre-configuration steps that must be done in order to ensure that the HA configuration is successful. 3.1 Configuring the /etc/hosts file: All the nodes in the cluster must have similar entries in the /etc/hosts file (as shown below) to ensure all hosts are mutually recognizable. Please make sure that the format of the entries as shown below: ! ! ! " " " # " " " " " " " $ $ $ $ $ $ $ /etc/hosts file contents for all nodes on cluster Entries for all networks for each database node should be exactly same on all nodes in the cluster. 3.2 Configuring the db2nodes.cfg file: Depending on the number of database partitions, the db2nodes.cfg file under the ~/sqllib/ directory must have the contents in the format shown below across all nodes. Typically in a D5100 Balanced Warehouse, the db2nodes.cfg file is present under the /db2home directory which is NFS shared from admin01: /shared_db2home on all the nodes. In the current implementation there are total 13 partitions: 1 on admin and 4 on each of the three data nodes. Hence the /db2home/bculinux/sqllib/db2nodes.cfg file looks like: " % & " " " " " " " " " " " db2nodes.cfg file contents 3.3 Configuring the /etc/services file All the nodes in the cluster must have the following entries in the /etc/services file to enable DB2 communication both inter and intra partitions. • The first entry below (db2c_bculinux 50001/tcp) corresponds to the port number that is used for external communication for the node. • The following entries (from DB2_bculinux 60000/tcp to DB2_bculinux_END 60012/tcp) correspond to the port numbers that are used for intra partition communication for a node. In the current example since the standby node would be configured to take over the admin and the 3 data nodes with 4 partitions each, the maximum number of partitions that would run on the standby would be 13. Hence 13 ports from 60000 to 60012 are reserved in this particular case. Also, since a BCU demands all the nodes must have the same configuration, the same 13 ports must be reserved on all nodes in the cluster. • Please ensure that all these ports numbers are unique in the /etc/services file and are not used for any other communication. ! ! ! ! ! ' ' ' ' ' ' ( ( (' (' ('" ('% & ) ) ) ) ") %) * * * * * * ! ! ! ! ! ! ! ! ' ' ' ' ' ' ' ' ('& (' (' (' (' (' (' ('+,! &) ) ) ) ) ) ) ) * * * * * * * * DB2 port settings in /etc/services 4. Configuring the Standby Node This section describes the settings that need to be done so that all the storage in the cluster is visible and available to the standby node as well as mountable in the event of a failover. Also, the settings that need to be done on the standby node so that it acts as the NFS server (in case the admin node goes down) and/or as an NFS client in case any node goes down. 4.1 NFS Server settings (configuring /etc/exports file) As mentioned before, in a D5100 Balanced Warehouse, the DB2 instance-owning admin node (admin01) acts as an NFS server for all the nodes in the cluster (including itself) which act as NFS clients. Typically there are two directories that are NFS shared across all the nodes: • /shared_db2home: The DB2 Instance home directory • /shared_home: User home directory for all non-DB2 users. The NFS server settings in a D5100 Balanced Warehouse are: rw,sync,fsid=X,no_root_squash Open the /etc/exports file of the admin node to confirm this. It should have entries like: In case the admin node goes down, the standby node must be able to take over as the NFS server for all the nodes. Hence, manually edit the /etc/exports file on the standby node so that it has entries identical to that of the admin node: 4.2 NFS Client settings (updating /etc/fstab file) Since all the nodes in the cluster (including admin node) act as NFS clients, the standby node must be configured to act as an NFS client in the event of a failover. The NFS client settings in a D5100 Balanced Warehouse are: rw,hard,bg,intr,suid,tcp,nfsvers=3,timeo=600,nolock Check the /etc/fstab file on all the nodes (including admin) for entries like these: ! " # ! $ % " # $ % Manually edit the /etc/fstab file on the standby node and add identical entries: ! " # ! $ % " # $ % Create the /db2home directory on standby and manually mount both /home and /db2home on standby. & % & & 4.3 Storage failover settings (updating /etc/fstab file) This section describes the settings that need to be done to ensure failover of the storage mount points to the standby node. First we’ll check whether all file systems are visible and available to standby and then we’ll configure standby so to make these mountable in the event of a failover. Verify that all the logical volumes from all the nodes are visible to standby: As root, issue the following command on standby: ' $! You should see a list of all the logical volumes from all the nodes in the HA group. In the current example, the output of the command is like: $ 01 * / * / / / / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 / 7 . -) * * * * * * *" *% *& * * * * . . ) 12 $ $ $ / / / / / / / / / / / / / / 7 / 7 ( 7 7 7 7 7 7 7 7 7 7 7 7 7 * * * * * * *" *% *& * * * * / 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 0# 3 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 666 666 &%% &%% &%% &%% &%% &%% &%% &%% &%% &%% &%% &%% &%% 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 # *5 / 0 *$5 7 / / 7 / 7 / 6 6 666 666 % 8 Logical volume setup on standby node Verify that all these logical volumes are available on standby: As root, issue the following command on standby: ' $! $ For every logical volume listed, you should see that the ’LV Status’ is listed as ’available’, as shown in the following sample output: 666 01 12 01 01 01 0 , , 99:! < # * 01 # 3 / 666 ) / /)/ 7 * ) / 7 * 7 * 2;<(6 7 68$$ ,=6$ ) / &%% " % 0+ <6#8> 6 $? @ 2 # A . . / &"- Logical volume details If any logical volume is marked as ’not available’, reboot the standby node and check again. Define the file systems on standby node and configure the /etc/fstab file so that it is able to mount respective storage in the event of a failover: Since all the admin and data nodes can failover to the standby node, all the file systems and /etc/fstab file on the standby node must be configured to be identical to that of admin and data nodes. In the current example we define the following file systems on standby: • Define the file systems for the DB2 instance home directory, the user home directory, and the NFS control directory (which are present in admin): % ( $ ) % % *+ • $ ) , !! $! , !! $! , !! $! $ ) )" $ )" ! $ ! $ )" $ $ ) ) ) ,-,-- ,-- Define the file system for staging space (present in admin): ) . . B) ) /)/ ) / ) ( " C C '( B DD ) )7 • Define file systems for database partitions 0 – 12 (present in admin and data nodes): ' % ( $ *+ , $ ) ./01 ) $ !! ) $! $ ) ./01 )" $ ) ,-- ' % ( $ (+ , $ ) ./01 ) $ !! ) $! $ ) ./01 )" $ ) ,-- ' % ( $ (+ , $ ! ! ) ./01 ) $ $! ) $ ) ./01 )" $ ) ,-- Similarly define the file systems for partitions 03 – 12 and add entries to /etc/fstab file. In the current implementation, the /etc/fstab has these added entries: ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ /)/ 7 ) / 7 . ) . 7 ) / 7 . ) . '. 7 ) / 7 / 7 )/ ) / ) ( 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 7 *") / 7 *" ) 7 7 *%) / 7 *% ) 7 7 *&) / 7 *& ) 7 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 7 * ) / 7 * ) 7 * ) / 7 * ) 7 * ) / 7 * ) -) . '. ). 7 -) . ' . ) . ' " ) ) ) ) ) ) ) ) ) ) 7 7 7 . ( " ( " C C 7 ( " C C C '( (),4!+ (),4!+ (),4!+ (),4!+ " (),4!+ % (),4!+ & (),4!+ (),4!+ (),4!+ (),4!+ ) (),4!+ ) (),4!+ ) (),4!+ C. C C C 7 C. C C C C ( ( ( ( ( ( ( ( ( ( C '( '( '( " C " C " C " C " C " C " C " C " C " C ( " ( " ( " C *C 7 / C C *C C C C C C C C C C C '( '( '( '( '( '( '( '( '( '( C C '( C C '( C C '( E"C E C 7 / E"C E C It is extremely important that the ‘noauto’ mount option is used for each of the ext3 file system in the /etc/fstab file. This would prevent system from auto-mounting the file system after reboot. TSA would take care of mounting the required file systems based on which nodes are up. For instance, initially admin and data nodes would have all their respective file systems manually mounted. In case data01 goes down, TSA will mount all its mount points that need to be transferred on standby. Once data01 comes up and standby is made to go offline, TSA will again ensure that the mount control is transferred back to data01. As part of the D5100 Balanced Warehouse configuration, admin and data nodes would already have this ‘noauto’ option set in their respective /etc/fstab files. In case not, please set the ‘noauto’ option in /etc/fstab/ across all nodes. 5. Configuring a DB2 Instance for HA using db2haicu Now that all the required pre-configuration is complete, we’ll configure the DB2 Instance for high availability using the db2haicu utility. Recall that the db2haicu utility can be run in two modes viz. the XML mode and the Interactive mode. This document would cover the configuration using the step by step interactive mode on the command line. 5.1 Procedure of running db2haicu 1. Prepare cluster nodes for db2haicu: Run the following command on all admin, data and standby nodes as root: ' " On a D5100 Balanced Warehouse, you can also use a single command from the mgmt node as ‘root’ which uses the cluster management system utility ‘dsh’. ' * " 2. Activate the database: On admin node, as instance owner (in this case bculinux), issue: & ! 23402 BCUDB is the database name used in the current implementation. 3. Run db2haicu: On Admin node, as instance owner, issue the db2haicu command. Once you run db2haicu, you’ll be prompted for inputs required for the HA configuration step by step. Below is the sample execution of the db2haicu used in the current implementation. Please note that in the current implementation, bond0 had been created using two network ports on the FCM network (on each node) and bond1 was created using two network ports on the Cluster network (on each node). Typically, this utility is run in-house i.e. before the system was shipped to the customer, so the corporate network isn’t available. Once the setup is delivered to the customer, additional configuration needs to be done to make the corporate network highly available which will be covered in the ‘Post Configuration steps’ section. Text in RED indicates user inputs. Text in BLUE indicates questions prompted by system/utility. Text in BLACK indicates information message by system/utility. [[email protected]]:~> db2haicu Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu). You can find detailed diagnostic information in the DB2 server diagnostic log file called db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains you create. For more information about configuring your clustered environment using db2haicu, see the topic called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information Center. db2haicu determined the current DB2 database manager instance is bculinux. The cluster configuration that follows will apply to this instance. db2haicu is collecting information on your current setup. This step may take some time as db2haicu will need to activate all databases for the instance to discover all paths ... When you use db2haicu to configure your clustered environment, you create cluster domains. For more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information Center. db2haicu is searching the current machine for an existing active cluster domain ... db2haicu did not find a cluster domain on this machine. db2haicu will now query the system for information about cluster nodes to create a new cluster domain ... db2haicu did not find a cluster domain on this machine. To continue configuring your clustered environment for high availability, you must create a cluster domain; otherwise, db2haicu will exit. Create a domain and continue? [1] 1. Yes 2. No 1 Create a unique name for the new domain: ha_domain Nodes must now be added to the new domain. How many cluster nodes will the domain ha_domain contain? 5 Enter the host name of a machine to add to the domain: admin01 Enter the host name of a machine to add to the domain: stdby01 Enter the host name of a machine to add to the domain: data01 Enter the host name of a machine to add to the domain: data02 Enter the host name of a machine to add to the domain: data03 db2haicu can now create a new domain containing the 5 machines that you specified. If you choose not to create a domain now, db2haicu will exit. Create the domain now? [1] 1. Yes 2. No 1 Creating domain ha_domain in the cluster ... Creating domain ha_domain in the cluster was successful. You can now configure a quorum device for the domain. For more information, see the topic "Quorum devices" in the DB2 Information Center. If you do not configure a quorum device for the domain, then a human operator will have to manually intervene if subsets of machines in the cluster lose connectivity. Configure a quorum device for the domain called ha_domain? [1] 1. Yes 2. No 1 The following is a list of supported quorum device types: 1. Network Quorum Enter the number corresponding to the quorum device type to be used: [1] 1 Specify the network address of the quorum device: 172.16.10.1 Refer to the appendix for details on quorum device Configuring quorum device for domain ha_domain ... Configuring quorum device for domain ha_domain was successful. The cluster manager found 10 network interface cards on the machines in the domain. You can use db2haicu to create networks for these network interface cards. For more information, see the topic 'Creating networks with db2haicu' in the DB2 Information Center. Create networks for these network interface cards? [1] 1. Yes 2. No 1 Enter the name of the network for the network interface card: bond0 on cluster node: admin01 1. Create a new public network for this network interface card. 2. Create a new private network for this network interface card. Enter selection: 2 Refer to the appendix below for more details Are you sure you want to add the network interface card bond0 on cluster node admin01 to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card bond0 on cluster node admin01 to the network db2_private_network_0 ... Adding network interface card bond0 on cluster node admin01 to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: bond0 on cluster node: data01 1. db2_private_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond0 on cluster node data01 to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card bond0 on cluster node data01 to the network db2_private_network_0 ... Adding network interface card bond0 on cluster node data01 to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: bond0 cluster node: data02 1. db2_private_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond0 on cluster node data02 to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card bond0 on cluster node data02 to the network db2_private_network_0 ... Adding network interface card bond0 on cluster node data02 to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: bond0 cluster node: data03 1. db2_private_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond0 on cluster node data03 to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card bond0 on cluster node data03 to the network db2_private_network_0 ... Adding network interface card bond0 on cluster node data03 to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: bond0 cluster node: stdby01 1. db2_private_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond0 on cluster node stdby01 to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card bond0 on cluster node stdby01 to the network db2_private_network_0 ... Adding network interface card bond0 on cluster node stdby01 to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: bond1 on on on on cluster node: stdby01 1. db2_private_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. Enter selection: 3 Create a separate private network for bond1 Are you sure you want to add the network interface card bond1 on cluster node data03 to the network db2_private_network_1? [1] 1. Yes 2. No 1 Adding network interface card bond1 on cluster node data03 to the network db2_private_network_1 ... Adding network interface card bond1 on cluster node data03 to the network db2_private_network_1 was successful. Enter the name of the network for the network interface card: bond1 on cluster node: data02 1. db2_private_network_1 2. db2_private_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond1 on cluster node data02 to the network db2_private_network_1? [1] 1. Yes 2. No 1 Adding network interface card bond1 on cluster node data02 to the network db2 _public_network_1 ... Adding network interface card bond1 on cluster node data02 to the network db2 _public_network_1 was successful. Enter the name of the network for the network interface card: bond1 on cluster n ode: data01 1. db2_private_network_1 2. db2_private_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond1 on cluster node data01 to the network db2_private_network_1? [1] 1. Yes 2. No 1 Adding network interface card bond1 on cluster node data01 to the network db2_private_network_1 ... Adding network interface card bond1 on cluster node data01 to the network db2_private_network_1 was successful. Enter the name of the network for the network interface card: bond1 on cluster node: admin01 1. db2_private_network_1 2. db2_private_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card bond1 on cluster node admin01 to the network db2_private_network_1? [1] 1. Yes 2. No 1 Adding network interface card bond1 on cluster node admin01 to the network db2_private_network_1 ... Adding network interface card bond1 on cluster node admin01 to the network db2_private_network_1 was successful. Retrieving high availability configuration parameter for instance bculinux ... The cluster manager name configuration parameter (high availability configuration parameter) is not set. For more information, see the topic "cluster_mgr - Cluster manager name configuration parameter" in the DB2 Information Center. Do you want to set the high availability configuration parameter? The following are valid settings for the high availability configuration parameter: 1.TSA 2.Vendor Enter a value for the high availability configuration parameter: [1] 1 Setting a high availability configuration parameter for instance bculinux to TSA. Now you need to configure the failover policy for the instance bculinux. The failover policy determines the machines on which the cluster manager will restart the database manager if the database manager is brought offline unexpectedly. The following are the available failover policies: 1. Local Restart -- during failover, the database manager will restart in place on the local machine 2. Round Robin -- during failover, the database manager will restart on any machine in the cluster domain 3. Mutual Takeover -- during failover, the database partitions on one machine will failover to a specific machine and vice versa (used with DPF instances) 4. M+N -- during failover, the database partitions on one machine will failover to any other machine in the cluster domain (used with DPF instances) 5. Custom -- during failover, the database manager will restart on a machine from a user-specified list Enter your selection: 4 You can identify mount points that are noncritical for failover. For more information, see the topic 'Identifying mount points that are noncritical for failover' in the DB2 Information Center. Are there any mount points that you want to designate as noncritical? [2] 1. Yes 2. No 2 The following DB2 database partitions can be made highly available: DB2 database partition number 0 DB2 database partition number 1 DB2 database partition number 2 DB2 database partition number 3 DB2 database partition number 4 DB2 database partition number 5 DB2 database partition number 6 DB2 database partition number 7 DB2 database partition number 8 DB2 database partition number 9 DB2 database partition number 10 DB2 database partition number 11 DB2 database partition number 12 Do you want to make all these DB2 database partitions highly available? [1] 1. Yes 2. No 1 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 0. Should the cluster node data01 be designated as an idle node for DB2 database partition 0? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 0? [1] 1. Yes 2. No 1 For all partitions we choose stdby01 as the idle node Should the cluster node data03 be designated as an idle node for DB2 database partition 0? [1] 1. Yes 2. No 2 Should the cluster node data02 be designated as an idle node for DB2 database partition 0? [1] 1. Yes 2. No 2 Adding DB2 database partition 0 to the cluster ... Adding DB2 database partition 0 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 0? [2] 1. Yes 2. No 2 For details on virtual IP, refer to the appendix M+N failover policy was chosen. You will need to specify the idle nodes for database partition 1. Should the cluster node admin01 be designated as an idle node for DB2 database partition 1? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 1? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 1? [1] 1. Yes 2. No 2 Should the cluster node data02 be designated as an idle node for DB2 database partition 1? [1] 1. Yes 2. No 2 Adding DB2 database partition 1 to the cluster ... Adding DB2 database partition 1 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 1? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 2. Should the cluster node admin01 be designated as an idle node for DB2 database partition 2? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 2? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 2? [1] 1. Yes 2. No 2 Should the cluster node data02 be designated as an idle node for DB2 database partition 2? [1] 1. Yes 2. No 2 Adding DB2 database partition 2 to the cluster ... Adding DB2 database partition 2 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 2? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 3. Should the cluster node admin01 be designated as an idle node for DB2 database partition 3? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 3? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 3? [1] 1. Yes 2. No 2 Should the cluster node data02 be designated as an idle node for DB2 database partition 3? [1] 1. Yes 2. No 2 Adding DB2 database partition 3 to the cluster ... Adding DB2 database partition 3 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 3? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 4. Should the cluster node admin01 be designated as an idle node for DB2 database partition 4? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 4? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 4? [1] 1. Yes 2. No 2 Should the cluster node data02 be designated as an idle node for DB2 database partition 4? [1] 1. Yes 2. No 2 Adding DB2 database partition 4 to the cluster ... Adding DB2 database partition 4 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 4? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 5. Should the cluster node data01 be designated as an idle node for DB2 database partition 5? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 5? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 5? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 5? [1] 1. Yes 2. No 2 Adding DB2 database partition 5 to the cluster ... Adding DB2 database partition 5 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 5? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 6. Should the cluster node data01 be designated as an idle node for DB2 database partition 6? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 6? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 6? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 6? [1] 1. Yes 2. No 2 Adding DB2 database partition 6 to the cluster ... Adding DB2 database partition 6 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 6? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 7. Should the cluster node data01 be designated as an idle node for DB2 database partition 7? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 7? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 7? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 7? [1] 1. Yes 2. No 2 Adding DB2 database partition 7 to the cluster ... Adding DB2 database partition 7 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 7? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 8. Should the cluster node data01 be designated as an idle node for DB2 database partition 8? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 8? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 8? [1] 1. Yes 2. No 1 Should the cluster node data03 be designated as an idle node for DB2 database partition 8? [1] 1. Yes 2. No 2 Adding DB2 database partition 8 to the cluster ... Adding DB2 database partition 8 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 8? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 9. Should the cluster node data01 be designated as an idle node for DB2 database partition 9? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 9? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 9? [1] 1. Yes 2. No 1 Should the cluster node data02 be designated as an idle node for DB2 database partition 9? [1] 1. Yes 2. No 2 Adding DB2 database partition 9 to the cluster ... Adding DB2 database partition 9 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 9? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 10. Should the cluster node data01 be designated as an idle node for DB2 database partition 10? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 10? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 10? [1] 1. Yes 2. No 1 Should the cluster node data02 be designated as an idle node for DB2 database partition 10? [1] 1. Yes 2. No 2 Adding DB2 database partition 10 to the cluster ... Adding DB2 database partition 10 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 10? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 11. Should the cluster node data01 be designated as an idle node for DB2 database partition 11? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 11? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 11? [1] 1. Yes 2. No 1 Should the cluster node data02 be designated as an idle node for DB2 database partition 11? [1] 1. Yes 2. No 2 Adding DB2 database partition 11 to the cluster ... Adding DB2 database partition 11 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 11? [2] 1. Yes 2. No 2 M+N failover policy was chosen. You will need to specify the idle nodes for database partition 12. Should the cluster node data01 be designated as an idle node for DB2 database partition 12? [1] 1. Yes 2. No 2 Should the cluster node admin01 be designated as an idle node for DB2 database partition 12? [1] 1. Yes 2. No 2 Should the cluster node stdby01 be designated as an idle node for DB2 database partition 12? [1] 1. Yes 2. No 1 Should the cluster node data02 be designated as an idle node for DB2 database partition 12? [1] 1. Yes 2. No 2 Adding DB2 database partition 12 to the cluster ... Adding DB2 database partition 12 to the cluster was successful. Do you want to configure a virtual IP address for the DB2 partition: 12? [2] 1. Yes 2. No 2 The following databases can be made highly available: Database: BCUDB Do you want to make all active databases highly available? [1] 1. Yes 2. No 1 Adding database BCUDB to the cluster domain ... Adding database BCUDB to the cluster domain was successful. All cluster configurations have been completed successfully. db2haicu exiting ... 4. Check status of the cluster: Once db2haicu exists, you can use the ‘lssam’ command to check the status of the cluster. The details on how to interpret output would be covered in the ‘Cluster monitoring’ section below. For now, just check that it shows ‘Online’ for all instance partitions and storage mount points on respective nodes and ‘Offline’ for all instance partitions and storage mount points on standby as illustrated below: [[email protected]]:~> lssam Online IBM.ResourceGroup:db2_bculinux_0-rg Nominal=Online |- Online IBM.Application:db2_bculinux_0-rs |- Online IBM.Application:db2_bculinux_0-rs:admin01 '- Offline IBM.Application:db2_bculinux_0-rs:stdby01 |- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0000-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0000-rs:admin01 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0000-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_10-rg Nominal=Online |- Online IBM.Application:db2_bculinux_10-rs |- Online IBM.Application:db2_bculinux_10-rs:data03 '- Offline IBM.Application:db2_bculinux_10-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0010-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0010-rs:data03 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0010-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_11-rg Nominal=Online |- Online IBM.Application:db2_bculinux_11-rs |- Online IBM.Application:db2_bculinux_11-rs:data03 '- Offline IBM.Application:db2_bculinux_11-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0011-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0011-rs:data03 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0011-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_12-rg Nominal=Online |- Online IBM.Application:db2_bculinux_12-rs |- Online IBM.Application:db2_bculinux_12-rs:data03 '- Offline IBM.Application:db2_bculinux_12-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0012-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0012-rs:data03 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0012-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_1-rg Nominal=Online |- Online IBM.Application:db2_bculinux_1-rs |- Online IBM.Application:db2_bculinux_1-rs:data01 '- Offline IBM.Application:db2_bculinux_1-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0001-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0001-rs:data01 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0001-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_2-rg Nominal=Online |- Online IBM.Application:db2_bculinux_2-rs |- Online IBM.Application:db2_bculinux_2-rs:data01 '- Offline IBM.Application:db2_bculinux_2-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0002-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0002-rs:data01 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0002-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_3-rg Nominal=Online |- Online IBM.Application:db2_bculinux_3-rs |- Online IBM.Application:db2_bculinux_3-rs:data01 '- Offline IBM.Application:db2_bculinux_3-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0003-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0003-rs:data01 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0003-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_4-rg Nominal=Online |- Online IBM.Application:db2_bculinux_4-rs |- Online IBM.Application:db2_bculinux_4-rs:data01 '- Offline IBM.Application:db2_bculinux_4-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0004-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0004-rs:data01 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0004-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_5-rg Nominal=Online |- Online IBM.Application:db2_bculinux_5-rs |- Online IBM.Application:db2_bculinux_5-rs:data02 '- Offline IBM.Application:db2_bculinux_5-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0005-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0005-rs:data02 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0005-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_6-rg Nominal=Online |- Online IBM.Application:db2_bculinux_6-rs |- Online IBM.Application:db2_bculinux_6-rs:data02 '- Offline IBM.Application:db2_bculinux_6-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0006-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0006-rs:data02 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0006-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_7-rg Nominal=Online |- Online IBM.Application:db2_bculinux_7-rs |- Online IBM.Application:db2_bculinux_7-rs:data02 '- Offline IBM.Application:db2_bculinux_7-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0007-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0007-rs:data02 '- Offline IBM.Application:db2mnt- db2fs_bculinux_NODE0007-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_8-rg Nominal=Online |- Online IBM.Application:db2_bculinux_8-rs |- Online IBM.Application:db2_bculinux_8-rs:data02 '- Offline IBM.Application:db2_bculinux_8-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0008-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0008-rs:data02 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0008-rs:stdby01 Online IBM.ResourceGroup:db2_bculinux_9-rg Nominal=Online |- Online IBM.Application:db2_bculinux_9-rs |- Online IBM.Application:db2_bculinux_9-rs:data03 '- Offline IBM.Application:db2_bculinux_9-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0009-rs |- Online IBM.Application:db2mntdb2fs_bculinux_NODE0009-rs:data03 '- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0009-rs:stdby01 5. Setting instance name in profiles.reg on standby: On the standby node verify that the registry file profiles.reg in the DB2 installation directory (/opt/IBM/dwe/db2/<version>) contains the name of the DB2 instance. If necessary, add the instance name to this file. 526 $ 789: ' $ 9 ) 6. Taking resource groups offline and online: Once the instance has been made highly available, you can use the ‘chrg –o <state>’ command to take the resource groups online and offline. For example, to take all the resource groups offline you can put the following commands in a file and run it as a script: * $ $ ) ( * $ $ ) ( * $ $ ) ( * $ $ ) "( * $ $ ) ;( * $ $ ) <( * $ $ ) #( * $ $ ) :( * $ $ ) =( * $ $ ) 8( * $ $ ) ( * $ $ ) ( * $ $ ) ( Similarly, the offline script for this environment would contain: * $ $ ) ( * $ $ ) ( * $ $ ) ( * $ $ ) "( * $ $ ) ;( * $ $ ) <( * $ $ ) #( * $ $ ) :( * $ $ ) =( * $ $ ) 8( * $ $ ) ( * $ $ ) ( * $ $ ) ( Although these commands return immediately, it takes some time for the resource groups to be brought online or offline. You can use the lssam command to monitor the status of the resource groups. You can also check the HA domains and the nodes that are part of respective domains by using the ‘lsrpdomain’ and ‘lsrpnode’ commands as show below: [[email protected]]:~> lsrpdomain Name OpState RSCTActiveVersion MixedVersions TSPort GSPort ha_domain Online 2.5.1.2 No 12347 12348 [[email protected]]:~> lsrpnode Name OpState RSCTVersion admin01 Online 2.5.1.2 stdby01 Online 2.5.1.2 data01 Online 2.5.1.2 data02 Online 2.5.1.2 data03 Online 2.5.1.2 5.2 Appendix for db2haicu Network Quorum device: A network quorum device is an IP address to which every cluster domain node can connect (ping) at all times. In the current implementation, the FCM network gateway IP is used assuming that as long as the FCM network segment is UP and RUNNING, the gateway will always be ping-able. No special software is required to be installed in this quorum device. It should just be reachable (ping-able) to all the nodes all the time. Public v/s Private address: In case the networks that you are trying to make highly available are private networks (internal to the Warehouse setup) like the FCM or the Cluster network, then you can choose to create a private network equivalency (e.g. db2_private_network_0). For public networks, i.e. the networks which the external applications use to connect to the setup, like the corporate network, you can choose to create a public network equivalency (e.g. db2_public_network_0) Virtual IP address: This is highly available IP address that the external clients/applications will use to connect to the database. Hence, this address should be configured on the same subnet which is exposed to the external clients/applications and only for database partition 0 on the administration BCU. Like in case of the current implementation, if db2haicu is run before putting the system on corporate network, configuration of virtual IP is not required. Once the system is put up on the corporate network, this additional configuration can be done by running db2haicu again. This is covered in the ‘Post configuration steps’ section. 6. Configuring the NFS Server for HA As mentioned earlier, the admin node in a D5100 Balanced Warehouse acts as an NFS server for all the other nodes (including itself). We will now see how to make this NFS server highly available to ensure that even if the admin node goes down, the NFS server keeps running on the standby node. Recollect that the two directories that are NFS shared across all nodes are /shared_db2home and /shared_home. Procedure 1. The NFS server would already be running on the admin node, so before re-configuring the NFS server for high availability, take it offline using the follow sequence of steps: a. Take the DB2 instance offline. b. Un-mount all of the NFS clients. (/db2home and /home) c. Take the NFS server offline. 2. Obtain an unused IP address in the same subnet (on the FCM network) as the admin node that will be used by the HA NFS server as a virtual IP address. In the current implementation we take the IP address 172.16.10.40 as the NFS server virtual IP. 3. Since TSA starts the services required to run the NFS server automatically, these services must be turned off on admin and standby nodes. In the event of a failure on the admin node, TSA would automatically start the NFS services on the standby node to avoid any downtime. If the NFS services start automatically at boot time, the failed node (admin) will attempt to restart another NFS server after it is restored, even though the NFS server is already running on the standby node. To prevent this situation from happening, we need to ensure that the NFS server does not automatically start at boot time on both admin and standby nodes by executing the following commands: % (($ ! $"< % (($ ! $"< % (($ ! $"< ! 4. There is a small file system on the lvnfsvarlibnfs logical volume that is created on the admin node during the initial setup phase of the D5100 Balanced Warehouse. On the admin node, mount this partition on /varlibnfs and copy all the files from /var/lib/nfs to this small file system. Then, un-mount the file system. 5. On the standby node create the /varlibnfs directory first. Then on both admin and standby node, back up the original /var/lib/nfs directory and create a link to the /varlibnfs mount point using the following commands: ' ! ! '$ * $ ! $ ! $ ! $ 9 $ 6. Verify that these conditions are still true for admin and standby nodes: • On both servers, the shared_home file system exists on /dev/vgnfs/lvnfshome and the shared_db2home file system exists on /dev/vgnfs/lvnfsdb2home. • On both servers, the /shared_home and /shared_db2home mount points exist. • On both servers, the /etc/exports file includes the following entries for the shared home directories: • • On both servers, the /etc/fstab file includes the following entries for the shared home directories: !! $! !! $! )" $ ) )" $ ) On both servers check for the following entry in the /etc/fstab file: !! $! ! $ ! $ )" $ ) 7. On all the NFS clients (all nodes), modify the /etc/fstab entries for the shared directories using the HA-NFS service IP address. We initially added the following entries: ! " ! # $ % " # $ % Modify these entries on all nodes so that they look like the following: : 9 #9 9; : 9 #9 9; ! " # ! " $ % # $ % 8. In the directory /usr/sbin/rsct/sapolicies/nfsserver on the admin node, edit the sanfsserver.conf file by changing the following lines. • In the nodes field, add the host names of both servers: # --list of nodes in the NFS server cluster nodes="admin01 stdby01" • Change the IP address to the virtual IP address and netmask of the NFS server used before: # --IP address and netmask for NFS server ip_1="172.16.10.40,255.255.255.0" • Add the network interface name used by the NFS server and host name of each server: # --List of network interfaces ServiceIP ip_x depends on. # Entries are lists of the form <network-interface-name>:<node-name>,... nieq_1="bond0:admin01,bond0:stdby01" • Add the mount points for the varlibnfs, shared_home and shared_db2home file systems: # --common local mountpoint for shared data # If more instances of <data_>, add more rows, like: data_tmp, data_proj... # Note: the keywords need to be unique! data_varlibnfs="/varlibnfs" data_work="/shared_db2home" data_home="/shared_home" • This configuration file must be identical on the admin node and the standby node. Therefore, copy this file over to the standby node: # scp sa-nfsserver.conf stdby01mgt:/usr/sbin/rsct/sapolicies/nfsserver 9. The sam.policies package comes with two versions of the nfsserver scripts. On both the admin node and the standby node, make the DB2 version of the script the active version using the following commands: # mv /usr/sbin/rsct/sapolicies/nfsserver/nfsserverctrl-server \ /usr/sbin/rsct/sapolicies/nfsserver/nfsserverctrl-server.original # cp /usr/sbin/rsct/sapolicies/nfsserver/nfsserverctrl-server.DB2 \ /usr/sbin/rsct/sapolicies/nfsserver/nfsserverctrl-server 10. On one of the servers, change to the /usr/sbin/rsct/sapolicies/nfsserver directory and then run the automatic configuration script to create the highly available NFS resources: # cd /usr/sbin/rsct/sapolicies/nfsserver # /usr/sbin/rsct/sapolicies/nfsserver/cfgnfsserver –p 11. Bring up the highly available NFS server: $ chrg -o Online SA-nfsserver-rg Although this command returns immediately, it takes some time for the NFS server to come online. Verify the status of the resource groups by issuing the ‘lssam’ command: After the resource groups have been brought online, your output for the SA-nfsserver-rg should look similar to the following: Online IBM.ResourceGroup:SA-nfsserver-rg Nominal=Online |- Online IBM.Application:SA-nfsserver-data-home |- Online IBM.Application:SA-nfsserver-data-home:admin01 '- Offline IBM.Application:SA-nfsserver-data-home:stdby01 |- Online IBM.Application:SA-nfsserver-data-varlibnfs |- Online IBM.Application:SA-nfsserver-data-varlibnfs:admin01 '- Offline IBM.Application:SA-nfsserver-data-varlibnfs:stdby01 |- Online IBM.Application:SA-nfsserver-data-work |- Online IBM.Application:SA-nfsserver-data-work:admin01 '- Offline IBM.Application:SA-nfsserver-data-work:stdby01 |- Online IBM.Application:SA-nfsserver-server |- Online IBM.Application:SA-nfsserver-server:admin01 '- Offline IBM.Application:SA-nfsserver-server:stdby01 '- Online IBM.ServiceIP:SA-nfsserver-ip-1 |- Online IBM.ServiceIP:SA-nfsserver-ip-1:admin01 '- Offline IBM.ServiceIP:SA-nfsserver-ip-1:stdby01 12. Manually mount the client NFS mount points on all servers. Verify that the /home and /db2home directories are mounted on both the admin node and standby node and that the /home and /db2home directories are readable and writable by each server. 13. To verify the configuration, use the following command to move the location of the NFS server from admin01 to stdby01: rgreq –o move SA-nfsserver-rg Verify that this command executes successfully. Issue the lssam command and verify that the NFS server resources are offline on admin01 and online on stdby01. Issue the same command to move the location of the NFS server from stdby01 back to admin01. 14. Create dependencies between the DB2 partitions and the NFS server by issuing the following commands from a script as the root user: # for DB2 resources: for each data partition 'x' do: mkrel -S IBM.Application:db2_bculinux_${x}-rs -G IBM.Application:SA-nfsserver-server -p DependsOnAny db2_bculinux_${x}-rs_DependsOn_SA-nfsserver-server-rel 15. Bring the DB2 instance back online and verify that all resources can start. 7. Post Configuration steps (at the customer site) Once the setup is delivered at the customer site and is put up on the corporate network, there are certain additional steps that must be undertaken to create public equivalency for this corporate network to make it highly available. You need to find an unused IP on the corporate network that would be used as a virtual IP (the highly available IP address which the external clients/applications would use to connect to the database). This additional setup does not require disturbing the initial setup which we did. You can run db2haicu again as the instance owner and just create the new equivalencies. Procedure 1. Run the db2haicu tool as the instance owner and select option 2. ’Add or remove a network interface.’ Do you want to add or remove network interface cards to or from a network? [1] 1. Add 2. Remove 1 Enter the name of the network interface card: eth1 Enter the logical port name of the corporate network of admin01 Enter the host name of the cluster node which hosts the network interface card eth1: admin01 Enter the name of the network for the network interface card: eth1 on cluster node: admin01 1. SA-nfsserver-nieq-1 2. db2_private_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: 3 We create a public network equivalency for the corporate network Are you sure you want to add the network interface card eth1 on cluster node admin01 to the network db2_public_network_0? [1] 1. Yes 2. No 1 Adding network interface card eth1 on cluster node admin01 to the network db2_public_network_0 ... Adding network interface card eth1 on cluster node admin01 to the network db2_public_network_0 was successful. Do you want to add another network interface card to a network? [1] 1. Yes 2. No 1 Enter the name of the network interface card: eth1 Enter the logical port name of the corporate n/w of stdby01 Enter the host name of the cluster node which hosts the network interface card eth1: stdby01 Enter the name of the network for the network interface card: eth1 on cluster node: stdby01 1. db2_public_network_0 2. SA-nfsserver-nieq-1 3. db2_private_network_0 4. Create a new public network for this network interface card. 5. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card eth1 on cluster node stdby01 to the network db2_public_network_0? [1] 1. Yes 2. No 1 Adding network interface card eth1 on cluster node stdby01 to the network db2_public_network_0 ... Adding network interface card eth1 on cluster node stdby01 to the network db2_public_network_0 was successful 2. We now need to configure the virtual IP which will be used as the highly available IP by the external applications and clients. This would be configured only on the database partition 0 (admin node). Find an unused IP address on the corporate network and run db2haicu as the instance owner. Select option 6. ’Add or remove an IP address.’ Do you want to add or remove IP addresses to or from the cluster? [1] 1. Add 2. Remove 1 Which DB2 database partition do you want this IP address to be associated with? 0 Enter the virtual IP address: 192.168.253.166 Enter the subnet mask for the virtual IP address 192.168.253.166: [255.255.255.0] 255.255.255.0 Select the network for the virtual IP 192.168.253.166: 1. db2_public_network_0 2. SA-nfsserver-nieq-1 3. db2_private_network_0 Enter selection: 1 Adding virtual IP address 192.168.253.166 to the domain ... Adding virtual IP address 192.168.253.166 to the domain was successful. Do you want to add another virtual IP address? [1] 1. Yes 2. No 2 3. Create dependencies between database partition 0 and the corporate network equivalency created before. Take the DB2 resources offline and then run the following command as the root user to create the dependency: mkrel -S IBM.Application:db2_bculinux_0-rs -G IBM.Equivalency:db2_public_network_0 -p DependsOn db2_bculinux_0-rs_DependsOn_db2_public_network_0-rel 4. Create the network quorum device for the corporate network. Run the db2haicu tool and select option 10. ’Create a new quorum device for the domain.’ Specify the gateway IP of the corporate network (in this case 192.168.253.1) This is how the network would look like after the above HA configuration: 8. Cluster monitoring This section describes how to interpret the output on the lssam command that we used before and how to monitor the cluster once it is configured. We will also discuss how the lssam output would indicate that a data node or an admin node has successfully failed over to the standby node. To explain how to interpret the lssam output, lets take a snippet of the one we got once our HA configuration was done. Online IBM.ResourceGroup:db2_bculinux_9-rg Nominal=Online |- Online IBM.Application:db2_bculinux_9-rs |- Online IBM.Application:db2_bculinux_9-rs:data03 '- Offline IBM.Application:db2_bculinux_9-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0009-rs |- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0009rs:data03 '- Offline IBM.Application:db2mnt-db2fs_bculinux_NODE0009rs:stdby01 The above snippet is of the database partition 9 which exists on node data03. The state in the first line indicates the overall state of the resource group db2_bculinux_9-rg. There are two resources that have been made highly available in this group: • • db2_bculinux_9-rs This is the DB2 database instance partition 9 that has been made highly available. If you see this resource is shown Online on data03 and Offline on stdby01 currently. This indicates that this instance partition is currently active on data03 which is up and running. db2mnt-db2fs_bculinux_NODE0009-rs This resource indicates the file system storage mount points that have been made highly available for database partition 9. Here again, these are currently mounted on data03 and hence the status is shown as Online on data03 and Offline on stdby01. In the event that data03 node goes down (to test, take it off the FCM network), TSA would try to bring up all the database instance partitions of data03 node and their associated storage on standy01. If you run the lssam command as soon as you take out the FCM network cables (both cables of bond0), you will see the following ‘lssam’ output: Pending Offline IBM.ResourceGroup:db2_bculinux_9-rg Nominal=Online |- Pending Offline IBM.Application:db2_bculinux_9-rs |- Pending Offline IBM.Application:db2_bculinux_9-rs:data03 '- Offline IBM.Application:db2_bculinux_9-rs:stdby01 '- Pending Offline IBM.Application:db2mnt-db2fs_bculinux_NODE0009-rs |- Pending Offline IBM.Application:db2mntdb2fs_bculinux_NODE0009-rs:data03 '- Offline IBM.Application:db2mnt-db2fs_bculinux_NODE0009rs:stdby01 The above output is shown only for the db2_bculinux_9-rg resource group. You will see a similar output for the other data03 resource groups db2_bculinux_10-rg, db2_bculinux_11-rg and db2_bculinux_12-rg as well. Soon after, the lssam output should show you a similar output for all db2_bculinux_9rg, db2_bculinux_10-rg, db2_bculinux_11-rg and db2_bculinux_12-rg resource groups Online IBM.ResourceGroup:db2_bculinux_9-rg Nominal=Online |- Online IBM.Application:db2_bculinux_9-rs |- Offline IBM.Application:db2_bculinux_9-rs:data03 '- Online IBM.Application:db2_bculinux_9-rs:stdby01 '- Online IBM.Application:db2mnt-db2fs_bculinux_NODE0009-rs |- Offline IBM.Application:db2mntdb2fs_bculinux_NODE0009-rs:data03 '- Online IBM.Application:db2mntdb2fs_bculinux_NODE0009-rs:stdby01 This now indicates that the database instance partition 9 and its associated storage points have been successfully started on stdby01. In case you have a database query which selects data from this partition, you could run the query to confirm that the data is successfully retrieved. Also, if you now open the ~/sqllib/db2nodes.cfg file, you will see its modified contents. All the instances of data03 would have been replaced by stdby01: " % & " " $ $ $ $ " $ $ $ $ Please note that in order bring up the resources back on data03, you will not only require to connect it back again on the FCM network, but also take stdby01 offline. In order to take corrective action on the node that had failed, it is mostly necessary to bring it up on the network and only once all the hardware/software re-configurations (in case needed) have been done can the resources be made active on that node. Hence resources do not automatically switch back to data03 unless standby is deliberately made offline. In case the admin node fails over to standby node then you would see the SA-nfsserverrg also Online on the standby node (and Offline on admin) along with db2_bculinux_0-rg: 4 : A F6 4 F6 4 F6 4 F6 4 G6 4 2 *-# 6 7 / 6 , E4 : ** -# 6 7 / 6 6. F6 477 : ** -# 6 7 / 6 6. G6 4 : ** -# 6 7 / 6 6. $ : ** -# 6 7 / 6 6/ 7 F6 477 : ** -# 6 7 / 6 6/ 7 G6 4 : ** -# 6 7 / 6 6/ 7 : ** -# 6 7 / 6 6 F6 477 : ** -# 6 7 / 6 6 G6 4 : ** -# 6 7 / 6 6 $ : ** -# 6 7 / 6 / F6 477 : ** -# 6 7 / 6 / G6 4 : ** -# 6 7 / 6 / $ : # / :>-# 6 7 / 6 *6 F6 477 : # / :>-# 6 7 / 6 *6 G6 4 : # / :>-# 6 7 / 6 *6 $ $ 4 : A F6 4 F6 4 2 *' : ** F6 477 : G6 4 : : ** F6 477 : G6 4 : ** ** ** ** (' 6 ' , E4 (' 6 ' ' 6 7 ' 6 6 (' 6 (' 6 $ (',4!+ 6 7 ' (',4!+ 7 ' (',4!+ 6 6 - $ 9. Lessons learnt during HA implementations Over various implementations we have came across few common observations which are cause of errors and configuration issues. Those are discussed here so that same can be avoided on the first attempt itself. 9.1 Hostname conflict Make sure that the OS hostnames on each node match with the hostnames mentioned in the db2nodes.cfg file. In case the hostnames do not match, you are likely to hit the following error while running db2haicu: > [[email protected]]" = . > $ ,", ,AB>>C /+0 6 5>>5.D,9> [[email protected]]>EBE1 = 9.2 Prevent auto-mount of shared file systems If you remember while adding entries in the /etc/fstab for all nodes, we emphasized on the “noauto” option. This option becomes very critical for the TSA to function smoothly and auto-mount file systems based on which node is up. In case “noauto” is not specified, the default option is ‘auto’ which means as soon as the failed node is re-booted, it will itself try to mount all file systems. This is undesirable since in most cases it is required to bring up the failed node before taking any corrective actions (hardware/software re-configurations). The “noauto” option prevents automounting of file systems at this stage and waits for TSA to mount them when the standby node is offlined. 9.3 Preventing file system consistency checks at boot time If you recollect the format of entries we used for /etc/fstab, you would notice that we had set the sixth field to zero (‘0’). /dev/vgdb2fsp0/lvdb2fsp0 /db2fs/bculinux/NODE0000 ext3 noauto,acl,user_xattr 1 0 /dev/vgdb2fsp1/lvdb2fsp1 /dev/vgdb2fsp2/lvdb2fsp2 /dev/vgdb2fsp3/lvdb2fsp3 /dev/vgdb2fsp4/lvdb2fsp4 /db2fs/bculinux/NODE0001 /db2fs/bculinux/NODE0002 /db2fs/bculinux/NODE0003 /db2fs/bculinux/NODE0004 ext3 ext3 ext3 ext3 noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr 1 1 1 1 0 0 0 0 /dev/vgdb2fsp5/lvdb2fsp5 /dev/vgdb2fsp6/lvdb2fsp6 /dev/vgdb2fsp7/lvdb2fsp7 /dev/vgdb2fsp8/lvdb2fsp8 /db2fs/bculinux/NODE0005 /db2fs/bculinux/NODE0006 /db2fs/bculinux/NODE0007 /db2fs/bculinux/NODE0008 ext3 ext3 ext3 ext3 noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr 1 1 1 1 0 0 0 0 /dev/vgdb2fsp9/lvdb2fsp9 /dev/vgdb2fsp10/lvdb2fsp10 /dev/vgdb2fsp11/lvdb2fsp11 /dev/vgdb2fsp12/lvdb2fsp12 /db2fs/bculinux/NODE0009 /db2fs/bculinux/NODE0010 /db2fs/bculinux/NODE0011 /db2fs/bculinux/NODE0012 ext3 ext3 ext3 ext3 noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr noauto,acl,user_xattr 1 1 1 1 0 0 0 0 This sixth field is used by the fsck (the file system check utility) to determine the order in which file systems should be checked. If not set to ‘0’, at boot time the fsck utility might attempt to check these file systems for consistency which is again undesirable. Consider the case when a data node fails over to standby. TSA would mount all the corresponding data node’s file systems on standby. Now, in case the sixth field in /etc/fstab for the data node is non-zero and it is re-booted, at boot time the linux fsck utility might attempt to check consistency of all its storage file systems, the control of which would be with standby. In such a case, the data node might not boot at all. Only workaround at this point would be to boot the data node in safe mode, logon as root and remove/comment all these entries in /etc/fstab. On the other hand, if the sixth field is set to ‘0’, the fsck does not attempt to check these file systems at boot time. This prevents such a situation from happening.

Set-Cookie: lastDocId=1617946; Path=/

Content-Language: en


markdown:
# Những mỏm đá "sống ảo" ở Việt Nam dân phượt nào cũng mê tít
**Đá Chồng ở Quảng Ninh**
Mỏm Đá Chồng ở xã Minh Thành, huyện Yên Hưng, tỉnh Quảng Ninh đang là địa điểm thu hút dân phượt. Nơi đây trở thành địa điểm tiếp theo để mọi người có thể “sống ảo” với những bức ảnh được chụp từ tảng đá chênh vênh trên mỏm núi.
*Đá Chồng được so sánh với phiến đá nổi tiếng Golden Rock ở Myanmar.*
Thoạt nhìn, nhiều người sẽ có cảm giác choáng ngợp bởi tảng đá này gồm nhiều phiến đá chồng lên nhau, mấp mé ở bờ vực. Có người còn so sánh Đá Chồng với phiến đá nổi tiếng Golden Rock - chùa Kyaiktiyo ở Myanmar.
Để đến được địa điểm này, du khách phải trải qua đoạn đường khá khó đi, đường rừng nhiều ngã rẽ và không có biển chỉ dẫn cụ thể. Do đó, bạn nên hỏi đường kỹ càng trước khi đi để tránh bị lạc.
Ngoài ra, dù bề mặt phiến đá đủ rộng để vài người cùng đứng nhưng bạn cũng cần cân nhắc thật kỹ trước khi có ý định lên Đá Chồng để “check-in”. Vì để có được bức hình “sống ảo”, bạn phải tự bám đá leo lên khá nguy hiểm.
**Mỏm đá trên đỉnh Pha Luông – Sơn La**
Từ lâu, rìa đá trên đỉnh Pha Luông đã trở thành “điểm hẹn” quen thuộc của những người mê xê dịch. Mất khoảng 3 tiếng đồng hồ để leo lên đến đỉnh nhưng bạn sẽ thấy công sức của mình xứng đáng khi đứng trên đỉnh Pha Luông. Không chỉ thu vào tầm mắt khung cảnh thiên nhiên hùng vĩ khoáng đạt mà bạn còn có cơ hội cho ra đời những bức ảnh đáng ghen tỵ khi ngồi tạo dáng ở “mỏm đá thần thánh” ở đây.
Dù mỏm đá chỉ nhô ra ngoài không quá nhiều nhưng ở nhiều góc chụp, có cảm giác như bạn đang ở rìa thế giới, cheo leo trên miệng vực. Vào mùa cao điểm, thậm chí mọi người còn phải… xếp hàng để có được một thời khắc chụp ảnh ở mỏm đá này.
*Mỏm đá trên đỉnh Pha Luông là nơi ra đời nhiều bức ảnh cực chất.* **Mỏm đá ở Thạch Thất – Hà Nội**
Dù chỉ cao hơn mặt đất… 5m nhưng mỏm đá ở Thạch Thất, Hà Nội vẫn là điểm đến được nhiều người yêu thích vì không cần di chuyển quá xa cũng có thể có những bức ảnh “sống ảo” cực chất.
*Chỉ cần chọn được góc chụp hợp lý là bạn có được bức ảnh "sống ảo" lung linh.*
Để đến được mỏm đá này, bạn cần leo đồi khoảng 1 giờ đồng hồ. Điểm khác biệt ở địa điểm này là du khách hoàn toàn có thể yên tâm tạo dáng chụp ảnh vì khoảng cách từ mỏm đá tới mặt đất khá gần. Chỉ cần chọn được góc máy đẹp và “diễn sâu” một chút là bạn có thể thu hết vào cảnh làng mạc, cây cối và cả những dãy núi xa xa như đang ngồi ở bên bờ vực thực sự.
**Moon** (tổng hợp) **Mọi phản ánh, ý kiến, thông tin nóng, bài vở cộng tác của độc giả có thể gửi cho chúng tôi theo địa chỉ:** **Email: banbientap@i-com.vn** **Hotline: 0914926900**
### Tin cùng chuyên mục
### Tin khác
[](/phap-luat/soc-khi-thieu-nu-bi-thanh-nien-moi-quen-tha-thuoc-kich-duc-dua-vao-nha-nghi-gio-tro-doi-bai-sau-hat-karaoke-2017011209242297.htm)
Sốc khi thiếu nữ bị thanh niên mới quen 'thả' thuốc kích dục đưa vào nhà nghỉ giở trò đồi bại sau hát karaoke
![Sốc khi thiếu nữ bị thanh niên mới quen "thả" thuốc kích dục đưa vào nhà nghỉ giở trò đồi bại sau hát karaoke Sốc khi thiếu nữ bị thanh niên mới quen "thả" thuốc kích dục đưa vào nhà nghỉ giở trò đồi bại sau hát karaoke](http://thumb.connect360.vn/unsafe/120x70/imgs.emdep.vn%2fShare%2fImage%2f2017%2f01%2f12%2fthieu-nu-bi-gio-tro-doi-bai-emdep-1-092449256.jpg)

WARC-Type: response

Content-Type: application/http; msgtype=response

WARC-Identified-Payload-Type: application/xhtml+xml

Connection: close

Cache-Control: private

Set-Cookie: SERVERID=emdep2; path=/

Content-Type: text/html; charset=utf-8


markdown:
**0904 227 453**
1.Quận
**Thanh Xuân**(trụ sở chính): 114 Khuất Duy Tiến - P.Nhân Chính - Q.Thanh Xuân - HN - Hotline: **02435510165**
2. Quận
**Long Biên - Hà Nội **(cửa hàng): 542 Nguyễn Văn Cừ - Hotline: **02432161795**
3. Quận
** Hà Đông - Hà Nội**(cửa hàng): 583 Quang Trung - Hotline: **02432009031**

WARC-Truncated: length

Server: nginx

Pragma: no-cache

Cache-Control: no-cache, pre-check=0, post-check=0

Set-Cookie: nv3c_Tzedj_nvvithemever=mw%2C%2C; expires=Sun, 14-Oct-2018 09:01:03 GMT; Max-Age=31104000; path=/; domain=.enterbuy.vn; httponly

Vary: Accept-Encoding

Content-Language: vi


markdown:
Em 2006, licenciei-me em Geologia, pela FCUL (Faculdade de Ciências da Universidade de Lisboa), movida pela paixão pelas ciências da Terra e da aventura de conhecer o mundo, com uma mochila ás costas.
Em 2009, fui mãe do T. e troquei, momentaneamente, as botas e a mochila, por biberons e fraldas!
Em 2010 cansados da estagnação em que a nossa vida se encontrava e pela falta de oportunidades arrumámos as tralhas. E fizémos 2600Km até ao Luxemburgo.
Encontrámos um país de florestas e de pastos sem fim! Com cheiro a terra, animais e a flores. Acordamos com o som das vacas e com o cantar dos passarinhos... Encontrámos um país, multicultural, com tantas diferenças. E também com muitos desafios! Mas estamos dispostos a arregaçar as mangas e fazer deste país a nossa casa.
e-mail:

WARC-Identified-Payload-Type: text/html

Server: GSE

X-Content-Type-Options: nosniff

Cache-Control: private, max-age=0

X-XSS-Protection: 1; mode=block

Content-Type: text/html; charset=UTF-8


markdown:
# Âm nhạc
-
[Multimedia](forums/multimedia.127/)
Download musics.
- Chủ đề:
- 161
Mới nhất:
- Bài viết:
- 8,812
[Nhạc nhảy HipHop cho anh em có nhu cầu](posts/27719015/) [ToanAudio](members/toanaudio.1731774/), 17/9/17
-
[House of fame](forums/house-of-fame.186/)
Tất cả các bài víêt hay sẽ được lưu lại tại đây.
- Chủ đề:
- 61
Mới nhất:
- Bài viết:
- 3,505
[[Guide] Hướng dẫn đăng ký và sử dụng Spotify tại Việt Nam](posts/19765091/) [iamgod](members/iamgod.255973/), 4/10/11
Trang 1 của 44 trang
[Tùy chọn hiển thị chủ đề](#)
Đang tải...
Trang 1 của 44 trang

Server: cloudflare-nginx

X-Frame-Options: SAMEORIGIN

X-Powered-By: centminmod


markdown:
# Putin chỉ trích hành động công bố cái chết của Gaddafi
Quốc tế
(GDVN) - Thủ tướng Nga Vladimir Putin hôm 26/10 đã lên tiếng lên án hành động cho công bố hình ảnh cái chết đẫm máu của nhà lãnh đạo Muammar Gaddafi.
[Putin, Medvedev lái máy gặt thu hoạch 12 tấn bắp ngô](/Quoc-te/Putin-Medvedev-lai-may-gat-thu-hoach-12-tan-bap-ngo-post18813.gd) [Thủ tướng Putin tiết lộ lý do ông sẽ trở lại điện Kremlin](/Quoc-te/Thu-tuong-Putin-tiet-lo-ly-do-ong-se-tro-lai-dien-Kremlin-post17939.gd) [Truyền thông Trung Quốc ngưỡng mộ Putin](/Quoc-te/Truyen-thong-Trung-Quoc-nguong-mo-Putin-post17818.gd) [Putin: Nga sẽ theo đuổi ngoại giao cân bằng](/Quoc-te/Putin-Nga-se-theo-duoi-ngoai-giao-can-bang-post17736.gd)
"Gần như toàn bộ gia đình Gaddafi đã bị giết chết. Hình ảnh thi thể của ông đã được đăng tải trên tất cả các kênh truyền hình trên thế giới. Không thể xem chúng mà không ghê tởm" - hãng thông tấn Interfax (Nga) dẫn lời phát biểu của Thủ tướng Putin tại một cuộc họp của Mặt trận nhân dân toàn Nga cho biết.
Đại tá Gaddafi đã bị bắt sống và bị giết chết hôm 20/10 tại thành phố Sirte - quê hương ông. Thi thể của ông sau đó được lưu trữ trong kho đông lạnh 3 ngày, nơi dân chúng có thể tới nhìn mặt cựu nhà lãnh đạo của họ lần cuối cùng.
Đại tá Gaddafi đã được chôn cất trong sáng sớm ngày 26/10 tại một địa điểm bí mật ở sa mạc.
Trong bối cảnh có nhiều ý kiến trái ngược nhau về hành động công bố công khai những hình ảnh và video ghi lại cảnh Đại tá Gaddafi bị bắt giữ, bị đánh đập, bị bắn trước khi chết và hình ảnh tại nơi lưu trữ xác của các phương tiện truyền thông quốc tế, Thủ tướng Nga đã bày tỏ ý kiến phản đối các hành động trên.
Thủ tướng Putin cho rằng: "Chẳng có gì tốt đẹp khi công bố những hình ảnh trên trên các phương tiện thông tin đại chúng".
Ông cũng cho rằng, không có tôn giáo nào trên thế giới ủng hộ việc các kênh truyền hình phát sóng cảnh giết hại một người đàn ông mình đầy máu và vẫn còn sống.
|
|![]()
Đại tá Gaddafi đã bị bắt sống và bị giết chết hôm 20/10 tại thành phố Sirte - quê hương ông. Thi thể của ông sau đó được lưu trữ trong kho đông lạnh 3 ngày, nơi dân chúng có thể tới nhìn mặt cựu nhà lãnh đạo của họ lần cuối cùng.
Đại tá Gaddafi đã được chôn cất trong sáng sớm ngày 26/10 tại một địa điểm bí mật ở sa mạc.
Trong bối cảnh có nhiều ý kiến trái ngược nhau về hành động công bố công khai những hình ảnh và video ghi lại cảnh Đại tá Gaddafi bị bắt giữ, bị đánh đập, bị bắn trước khi chết và hình ảnh tại nơi lưu trữ xác của các phương tiện truyền thông quốc tế, Thủ tướng Nga đã bày tỏ ý kiến phản đối các hành động trên.
Thủ tướng Putin cho rằng: "Chẳng có gì tốt đẹp khi công bố những hình ảnh trên trên các phương tiện thông tin đại chúng".
Ông cũng cho rằng, không có tôn giáo nào trên thế giới ủng hộ việc các kênh truyền hình phát sóng cảnh giết hại một người đàn ông mình đầy máu và vẫn còn sống.
Nguyễn Hường (Theo Rian)
Putin chỉ trích hành động công bố cái chết của Gaddafi
![](http://img.giaoduc.net.vn/Uploaded/nguyenhuong/2011_10_27/putin.jpg)
[Tổng thống Putin](/chu-de/Tong-thong-Putin/421.gd)

Cache-Control: no-cache


markdown:
Trang Chủ
Giáo Phận
Giới Thiệu-Địa Danh
Các Giáo Xứ-Miền Kontum
Các Giáo Xứ-Miền Gialai
Liên Hệ
Tin Tức
Giáo Phận KonTum
Giáo Hội Việt Nam
Giáo Hội Hoàn Vũ
Sinh Hoạt Giáo Phận
Tin Tức Tổng Hợp-Blog
Đức Mẹ Măng Đen
Lịch sử Hình Thành
Hành Hương Kính Mẹ
Thông Báo
Thư Mục Vụ Giám Mục
Văn Phòng Tòa Giám Mục
Lịch Tĩnh Tâm
Thông Báo Khác
Phụng Vụ
Suy Ngẫm Lời Chúa
Lời Chúa Hằng Ngày
Phụng Vụ Các Giáo Xứ
Bài Giảng
Các Bài Giảng
Audio-Video Bài Giảng
Linh Mục-Liên Tu Sĩ
Hội Đồng Linh Mục
Đức Giám Mục
Linh Mục
Tu Sĩ
Mục Vụ-Đoàn Thể
Ban Mục Vụ - Hôn Nhân Gia Đình
Đoàn Thể - Sinh Viên - Các Ban
Văn Hóa-Văn Thơ
Giới Thiệu Nhạc
Giới Thiệu Sách-Thơ
Tư Liệu
Hình Ảnh
Video
Sách-Bài Viết
N, 00-00-2011

00 :00:00
Bản Tin Giáo Phận Kontum Số 01 Và Số 2 Năm 2017
(04:03:2017 |07:40 AM)
Ban mục vụ Truyền thông Giáo phận Kontum xin giới thiệu Bản tin số 01 và số 2 năm 2017
GPKONTUM (04/03/2017) KONTUM
XIN KÍNH MỜI
Các Tin Tức - Bài Viết Khác:
(12:09:2017 | 02:59 AM)
Đại Hội Legio Mariae Miền Pleiku
(12:09:2017 | 02:56 AM)
70 Năm Nhìn Lại
(12:09:2017 | 02:50 AM)
Legio Marie Comitium Pleiku Mừng Sinh Nhật Đức Mẹ
(12:09:2017 | 02:48 AM)
Thuyết Trình 5: Đào Tạo Giáo Lý Viên Chứng Nhân Niềm Vui Tin Mừng
(12:09:2017 | 02:32 AM)
Sửa Lỗi Huynh Đệ
(01:09:2017 | 07:05 AM)
Thánh Lễ Bổ Nhiệm Linh Mục Chánh Xứ Giáo Xứ La-Sơn
(01:09:2017 | 07:01 AM)
Tu sĩ Phê-rô Maria Nguyễn Phúc Linh Được Chúa Gọi Về Lúc: 08h00 Chúa Nhật, Ngày 27/08/2017
(26:08:2017 | 09:29 AM)
Thư Chúc Mừng Đức Cha Louis Nguyễn Anh Tuấn Của Ủy Ban Mục Vụ Gia Đình
(26:08:2017 | 09:20 AM)
Đức Thánh Cha Phanxicô Bổ Nhiệm Giám Mục Phó Giáo Phận Long Xuyên
(26:08:2017 | 09:13 AM)
Đức Thánh Cha Phanxicô Bổ Nhiệm Tân Giám Mục Phụ Tá Thứ Hai Cho Tổng Giáo Phận Sài Gòn
(26:08:2017 | 09:01 AM)
Thư Ngỏ Của Uỷ Ban Giáo Lý Đức Tin
(26:08:2017 | 08:56 AM)
Chìa Khóa Vương Quốc
(25:08:2017 | 07:55 AM)
Thánh Lễ Nhận Xứ Giáo Xứ La Sơn, hạt Mang Yang
(22:08:2017 | 08:06 AM)
Ngày Họp Mặt Tháng 9.2017: Theo Giáo Hạt
(22:08:2017 | 07:51 AM)
ĐTC Phanxicô Gửi Các Tham Dự Viên Buổi Hội Thảo Quốc Tế Về Việc Dạy Giáo Lý 2017
(22:08:2017 | 07:43 AM)
Tu Đoàn Bác Ái Xã Hội Phan Thiết Tham Dự Khóa Thường Huấn Hè 2017
(19:08:2017 | 08:06 AM)
Ứng Dụng Laudato Si Vào Đời Sống Trong Giáo phận
(18:08:2017 | 04:15 PM)
8 Thầy Phó Tế Cho Thiên Chức Linh Mục Sắp Tới
(18:08:2017 | 04:08 PM)
Đức Tin Làm Cho Sống
(13:08:2017 | 10:17 PM)
Thánh Lễ Cầu Nguyện Cho Bà Cố Maria
Trang :
1
-
2
-
3
-
4
-
5
-
6
-
7
-
8
-
9
-
10
Điểm Nhấn
Hội Nghị Thường Niên Kỳ II-2017 Hội Đồng Giám Mục Việt Nam..
70 Năm Nhìn Lại..
8 Thầy Phó Tế Cho Thiên Chức Linh Mục Sắp Tới..
Thánh Lễ Tiên Khấn Lần Đầu Của 11 Yă Hội Dòng Ảnh Phép Lạ Kont..
Chương Trình Điều Phối Việc Dạy Giáo Lý Trong Giáo Phận Kontum..
Tìm Tin Tức - Bài Viết
Tin Tức - Bài Viết Mới
-
Trực Tuyến Thánh Lễ Truyền Chức Giám Mục Luy Nguyễn Anh Tuấn
-
Biên Bản Hội Nghị Thường Niên Kỳ II-2017 Hội Đồng Giám Mục Việt Nam
-
Hội Nghị Thường Niên Kỳ II-2017 Hội Đồng Giám Mục Việt Nam
-
Tá Điền Sát Nhân
-
Văn Phòng TGM xin thông báo
-
Thánh Lễ Truyền Chức Linh Mục, Giáo Phận Kon Tum, 29-09-2017- Phần II
-
Thánh Lễ Truyền Chức Linh Mục, Giáo phận Kontum Phần I
-
Tác Động Tiêu Cực Của Smartphone Trên Thế Hệ Trẻ
-
Hạnh Động Tuyên Xưng
-
Giải Viết Văn Đường Trường Năm 2017 Tại Làng Sông Qui Nhơn
-
Bản Tin Số 7 Và Số 8 Năm 2017
-
Thánh Lễ Nhận Chính Xứ Plei Ngol Khóp Của Cha Philipphê Đỗ Đức Hoan SDB
-
Thiệp Mời Phong Chức Linh
-
Đức Thánh Cha Phanxicô Ban Hành Tự Sắc Về Vấn Đề Dịch Các Bản Văn Phụng Vụ
-
Thánh Lễ Tạ Ơn Ngày Truyền Thống Comitium Miền Pleiku
Album Ảnh
Bài Giảng
Tiệc Cưới Và Áo Cưới...
Tá Điền Sát Nhân...
Hạnh Động Tuyên Xưng...
Sửa Lỗi Huynh Đệ...
Chìa Khóa Vương Quốc...
Video
Trực Tuyến Thánh Lễ Truyền Chức Giám Mục Luy Nguyễn Anh Tuấn..
Thánh Lễ Truyền Chức Linh Mục, Giáo Phận Kon Tum, 29-09-2017- Phần II..
Thánh Lễ Truyền Chức Linh Mục, Giáo phận Kontum Phần I..
Thánh Lễ Tạ Ơn Ngày Truyền Thống Comitium Miền Pleiku..
Đại Hội Legio Mariae Miền Pleiku..
Thánh Lễ Bổ Nhiệm Linh Mục Chánh Xứ Giáo Xứ La-Sơn..
Thánh Lễ Cầu Nguyện Cho Bà Cố Maria..
Lễ Trao Giải Hoa Núi Rừng III Năm 2017 – Giáo Phận Kontum..
Hội Trại Sinh Viên Công Giáo 2017- Giáo Phận Kontum..
Thánh Lễ Tạ Ơn Bế Giảng Khóa Sư Phạm Giáo Lý Giáo Hạt Chư Prông 29.07.20..
Thánh Lễ Tạ Ơn Tiên Khấn Dòng Ảnh Phép Lạ Kontum 3.8.2017 - Phần II..
Thánh Lễ Tạ Ơn Tiên Khấn Dòng Ảnh Phép Lạ – 3.08.2017..
Khóa Huấn Luyện Huynh Trưởng Giáo Lý Viên Giáo Hạt Chư Prong – Gp. Kontu..
Học Tiếng Nước Ngoài
-Học tiếng Do Thái
-Học tiếng Hy Lạp
-Học tiếng Nhật
-Thành ngữ
-Tiếng Anh Online
-Tiếng Anh theo BBC
Liên Kết
Website:
---Chọn Liên Kết---
Hội Đồng Giám Mục Việt Nam
Vatican
Vatican-Việt Nam
Liên HĐGM Á châu
Hội Linh Mục Xuân Bích
Tổng Giáo Phận Huế
Tổng Giáo Phận Hà Nội
Giáo phận Thái Bình
Giáo phận Thanh Hoá
Giáo phận Lạng Sơn
Giáo phận Vinh
Giáo phận Đà Lạt
Uỷ ban BAXH - Caritas Việt Nam
Giáo phận Ban Mê Thuột
Tổng Giáo Phận TP HCM
Uỷ ban Kinh Thánh
Uỷ ban Mục vụ Di dân
Uỷ ban Giáo Lý Đức Tin
Dòng Phanxicô Việt Nam
Dòng Kytô Vua
Trung tâm Thánh Mẫu Toàn Quốc La Vang
Website LM Gioan Nguyễn Lợi
Website Tin Mừng
Phút Hồi Tâm
Thánh Linh
Nam Giao Thi Ca (Thơ Công Giáo)
Nhạc Thánh
Lam Hồng
Sinh Viên Công Giáo Hoàng Xá
Kinh Thánh cho mọi người
Quý vị xem thêm các tin tức và bài viết trước tháng 6 năm 2014
tại đây
Trang Blog Giáo Phận KonTum
http://gpkontum.wordpress.com
Website được thiết kế và phát triển bởi Truyền Thông Giáo Phận KonTum.
Website hiển tốt nhất với trình duyệt "Firefox","Google Chrome" và độ phân giải 1024x768
Mọi trích dẫn hoặc đăng lại từ trang này mong quý vị ghi rõ "Trích Từ Nguồn GiaoPhanKonTum". Xin Thiên Chúa Ban Bình An Cho Quý Vị


markdown:
Thông tin liên quan
Tây Du Ký
Tác giả : Ngô Thừa Ân
Thể loại : Truyện Audio
Đọc truyện : Bạch Minh
Quốc gia : Trung Quốc
Nguồn : Truyen.blogspot.com
Người đăng : Hồ Vũ Tuấn Anh
Ngày đăng : 21/12/2010
Hồi 54
Xem tiếp : Hồi 55


markdown:
[Vụ 6.000 quả trứng](http://vnexpress.net/Vietnam/Xa-hoi/2006/03/3B9E7FAD/) gà Trung Quốc vừa bị các cơ quan chức năng TP HCM phát hiện, khiến người Sài Gòn chợt giật mình khi đi mua sắm. Khách đi chợ từ chối mua sản phẩm gia cầm vì sợ mua phải hàng nhập khẩu.
Chị Minh Thư, chuyên bán trứng ở chợ Văn Thánh, sáng nay phải bắt ghế túc trực trước gian hàng để luôn miệng mời chào khách. Nhác thấy bà Hồng Liên, một khách quen thuộc của mình, chị Thư đon đả chạy lại cầm tay. Song, bà Liên lắc đầu quầy quậy: "Sợ bị đụng trứng Trung Quốc quá đi", mặc chị Thư thề sống chết là hàng Việt Nam 100%. Đến 10h sáng nay, chị chỉ mới bán được 2 quả trứng gà, trong khi mọi hôm, vào giờ này chị đã tiêu thụ hết ít nhất 200 quả trứng.
Sức tiêu thụ thịt trứng ở ngoại thành không đến nỗi vắng vẻ như nội thành, nhưng theo bà Vương, bán trứng tại chợ An Lạc, huyện Bình Chánh: "Nhiều người hỏi có phải trứng Trung Quốc giá rẻ bị cúm không, làm tôi thay vì phải hạ giá vì ế thì lại tăng lên 100-200 đồng/trứng để chứng tỏ hàng tốt của Việt Nam".
![]()
|Khó phân biệt được trứng gà nào là của Trung Quốc. |Ảnh: P.A.
Hàng lậu đổ vào chỗ trũng
Những biện pháp gắt gao, thắt chặt kiểm soát đàn gia cầm để ngăn chặn dịch cúm lây lan, khiến nguồn cung trứng gia cầm trong nước những tháng qua trở nên khan hiếm. Đến nay, giá thịt gà tam hoàng đã ở mức 40.000-45.000 đồng/kg làm sẵn, còn gà ta đã tăng đến kỷ lục 70.000-75.000 đồng/kg thịt hơi (chưa làm lông). Thị trường hầu như vắng bóng thịt gà tơ, nhỏ hơn 1kg mà chỉ bán thịt gà già, có trọng lượng gần 2 kg trở lên.
Trứng gà, vịt cũng không vượt ra ngoài ngoại lệ. Trứng vịt gần như không mua được hàng ngoài thị trường, còn trứng gà liên tục tăng giá từ 800 đồng/trứng, lên 1.000 rồi 1.200, hiện nay là 1.500 đồng/trứng. Sự khan hiếm của thị trường đã tạo điều kiện cho gà lậu từ biên giới các tỉnh lũ lượt đổ về TP HCM.
Phó chủ tịch UBND huyện Châu Thành, tỉnh Long An Trương Văn Biết nói, mới đây huyện này đã phải đóng cửa biên giới giáp Campuchia đối với các loại sản phẩm gia cầm để ngăn chặn hàng lậu, nhiễm bệnh tràn vào nội địa. Đại diện Chi cục Thú y TP HCM cũng cho rằng, hiện 3 cửa ngõ chính cho hàng gia cầm lậu từ nước ngoài vào thành phố là khẩu Mộc Bài (Tây Ninh), Mộc Hóa (Long An) và các tỉnh biên giới phía Bắc.
Tuần qua, Bộ trưởng Nông nghiệp phát triển nông thôn Cao Đức Phát cũng đã tổ chức chuyến khảo sát tình hình nhập lậu gia cầm từ Trung Quốc sang tại các tỉnh biên giới phía Bắc. Theo kết luận của đoàn kiểm tra, hiện có sự chênh lệch giá cả rất lớn giữa gà Trung Quốc và Việt Nam tại tỉnh Lạng Sơn. Đơn cử, giá thịt gà Trung Quốc chỉ 3 nhân dân tệ một kg, tương đương với 6.000 đồng, mang về Lạng Sơn bán 25.000-30.000 đồng/kg, tức lời gấp 5 lần giá gốc. Lạng Sơn cũng đã phát hiện được hơn 12 tấn thịt gà nhập lậu từ Trung Quốc, mang tiêu hủy.
Một chuyên gia chăn nuôi tháp tùng đoàn, cho biết, còn có hàng trăm nghìn trứng gà Trung Quốc đang được tiêu thụ ở Lạng Sơn. Chuyên gia này cũng khẳng định, không ít thịt, trứng gà Trung Quốc đã được các thương lái chuyển vào các tỉnh phía Nam, theo quy luật "chỗ nào trũng nhất thì nước chảy vào".
Phòng bệnh hơn chữa bệnh
Trao đổi với
*VnExpress*, Cục trưởng Chăn nuôi Nguyễn Đăng Văng khẳng định: "Cho đến giờ này chưa thể kiểm tra được thịt, trứng gà Trung Quốc có nhiễm bệnh cúm hay không, tuy nhiên không loại trừ 2 khả năng, một là gà thải từ Trung Quốc, hoặc do nước này cũng đang có dịch bệnh nên người dân không ăn thịt, trứng, giá rẻ khiến nhiều người tuồn hàng sang Việt Nam bán kiếm lời".
Trong khi đó, một chuyên gia thú y tại TP HCM khuyến cáo: "Không loại trừ trường hợp thịt, trứng Trung Quốc có mang mầm bệnh cúm, gây lây lan dịch". Tuy nhiên, theo chuyên gia này, rất khó phân biệt được về mặt cảm quan, xuất xứ cũng như nguồn gốc trứng. "Nhìn bên ngoài không thể phân biệt được thịt, trứng nào là của Trung Quốc. Chúng tôi cũng bó tay, trừ khi chủ hàng khai nhận", chuyên gia này cho biết. Còn nhiều người tiêu dùng thì cho rằng, có thành dịch hay không thì chưa biết, nhưng tốt hơn hết là không nên ăn trứng, thịt gà Trung Quốc.
Ông Văng cho hay, các trường hợp thịt, trứng Trung Quốc từng bị phát hiện, bằng mắt quan sát, không thể phân biệt được xuất xứ hàng. Theo ông Văng, hiện Bộ Nông nghiệp phát triển nông thôn đã đề xuất Chính phủ thành lập các đoàn kiểm tra, xác định mầm bệnh, lấy mẫu kiểm tra. Đồng thời, Lạng Sơn và 6 tỉnh biên giới phía Bắc là Bắc Giang, Bắc Ninh, Tuyên Quang, Phú Thọ, Vĩnh Phúc, Hưng Yên phải tập trung kiểm soát hàng lậu chặt chẽ, ngăn chặn tình trạng đầu nậu buôn vận chuyển hàng qua biên giới từ các chợ nằm ngoài đường biên.
"Song, theo tôi, hiện nay công tác kiểm soát rất khó đảm bảo vì các tỉnh biên giới phía Bắc kéo dài đến 250 km, trong khi lực lượng kiểm tra rất mỏng. Cái chính vẫn là công tác tuyên truyền trong dân chúng để không buôn bán thịt, trứng Trung Quốc qua biên giới", ông Văng nhấn mạnh.
Chi cục trưởng quản lý thị trường Huỳnh Tấn Phong cũng cho biết, sau vụ trứng lậu từ Trung Quốc bị phát hiện 2 ngày trước, chi cục cũng đã có chỉ đạo sẽ tổ chức, kiểm tra gắt gao hơn nữa để ngăn chặn dịch bệnh có thể lây lan, trong trường hợp sản phẩm có mang mầm bệnh.
**Phan Anh**
▪
[Người phát ngôn Bộ Ngoại giao VN: Quyết định của EU là không công bằng](/kinh-doanh/nguoi-phat-ngon-bo-ngoai-giao-vn-quyet-dinh-cua-eu-la-khong-cong-bang-425558) * (25/03/2006)*
▪
[Ngân hàng Đông Á: Ra mắt dịch vụ "Mở thẻ lấy ngay"](/kinh-doanh/ngan-hang-dong-a-ra-mat-dich-vu-mo-the-lay-ngay-425557) * (25/03/2006)*
▪
[Hầm chui M1 bị xô lệch do xe tải lưu thông?](/kinh-doanh/ham-chui-m1-bi-xo-lech-do-xe-tai-luu-thong-425556) * (25/03/2006)*
▪
[Đất treo, dân nhịn](/kinh-doanh/dat-treo-dan-nhin-425555) * (25/03/2006)*
▪
[Thuê bao đăng ký mới S-Fone tăng kỷ lục](/kinh-doanh/thue-bao-dang-ky-moi-s-fone-tang-ky-luc-425554) * (25/03/2006)*
▪
[Thành lập Hội đồng nghiệm thu Nhà nước các công trình xây dựng](/kinh-doanh/thanh-lap-hoi-dong-nghiem-thu-nha-nuoc-cac-cong-trinh-xay-dung-425553) * (25/03/2006)*
▪
[Xung quanh biểu thuế nhập khẩu ôtô cũ: Những phản ứng trái chiều](/kinh-doanh/xung-quanh-bieu-thue-nhap-khau-oto-cu-nhung-phan-ung-trai-chieu-425552) * (25/03/2006)*
▪
[Doanh nghiệp ma: Có sự tiếp tay của cơ quan cấp phép, thuế...](/kinh-doanh/doanh-nghiep-ma-co-su-tiep-tay-cua-co-quan-cap-phep-thue-.-425551) * (25/03/2006)*
▪
[Nói tiếp về chặt, trồng?](/kinh-doanh/noi-tiep-ve-chat-trong-425549) * (25/03/2006)*
▪
[Doanh nghiệp FDI không muốn đăng ký lại](/kinh-doanh/doanh-nghiep-fdi-khong-muon-dang-ky-lai-425490) * (25/03/2006)*


markdown:
-
[Thiết bị vệ sinh](/thiet-bi-ve-sinh.html)
-
[Thiết bị nhà bếp](/thiet-bi-bep.html)
-
[Gạch ốp lát](/gach-op-lat.html)
-
[Gạch Đồng Tâm](/gach-dong-tam.html)
-
[Gạch Bạch Mã](/gach-bach-ma.html)
-
[Gạch Prime](/gach-prime.html)
-
[Gạch TAICERA](/gach-taicera.html)
-
[Taicera TKG](/taicera-tkg.html)
-
[Gạch ROYAL](/gach-royal.html)
-
[Taicera KERABEN](/taicera-keraben.html)
-
[Gạch CATALAN](/gach-catalan.html)
-
[Gạch KIS](/gach-kis.html)
-
[Gạch MIKADO](/gach-mikado.html)
-
[Gạch Thạch Bàn](/gach-thach-ban.html)
-
[Gạch Hoàn Mỹ](/gach-hoan-my.html)
-
[Gạch Trung Quốc](/gach-trung-quoc.html)
-
[Gạch Viglacera](/gach-viglacera.html)
-
[Gạch VietCeramics](/gach-vietceramics.html)
-
[Gạch Tây Ban Nha](/gach-tay-ban-nha.html)
-
[Gạch INDOGRESS](/gach-indogress.html)
-
[Gạch lát sân vườn](/gach-lat-san-vuon.html)
-
[Gạch ốp bể bơi](/gach-op-be-boi.html)
-
[Gạch kính lấy sáng](/gach-kinh-lay-sang.html)
-
[Gạch ốp theo bộ](/gach-op-theo-bo.html)
-
[Gạch lát vỉa hè](/gach-lat-via-he.html)
-
-
[Thiết bị theo mùa](/thiet-bi-theo-mua.html)
-
[Sàn gỗ công nghiệp](/san-go-cong-nghiep.html)
-
[Điện gia dụng](/dien-gia-dung.html)
-
[Sản phẩm khác](/san-pham-khac.html)

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0


markdown:
[Nhận định](/nhan-dinh-bong-da.html) [Tường thuật](/tin-tuc/tuong-thuat-bong-da) [Tin tức bóng đá](/tin-tuc-bong-da.html) [Kết quả bóng đá](/ket-qua-bong-da.ikq) [Lịch thi đấu bóng đá](/lich-thi-dau-bong-da.hs) [Kèo bóng đá](/ty-le-keo-bong-da.html) [Bảng xếp hạng bóng đá](/bang-xep-hang-bong-da) [Video Clip](/video-clip-bong-da.html) [Ảnh bóng đá](/anh.abd) [Game dự đoán](/du-doan-bong-da.html)
Quảng cáo của đối tác
[Premier Village Phú Quốc](http://lotuslandvn.com/du-an/premier-village-phu-quoc/) [xổ số mega](http://www.xosomega.com/) [soi cầu lô tô](http://soicauxoso.com/soi-cau-lo-to) [game online](http://lagavi.com/tin-tuc/game-online) [soi cầu đặc biệtt](http://soicauxoso.com/soi-cau-dac-biet) [xổ số mega 6/45](http://www.xosomega.com/645/) [mua ban nha dat](http://timdat.net/) [nha dat da nang](http://timdat.net/nha-dat-da-nang) [nha dat ha noi](http://timdat.net/nha-dat-ha-noi) [Premier Village Phú Quốc Resort](http://lotuslandvn.com/tag/premier-village-phu-quoc/) [xổ số tự chọn](http://www.xosomega.com/tag/xo-so-mega-645/) [nha dat hai duong](http://timdat.net/nha-dat-hai-duong) [nha dat hai phong](http://timdat.net/nha-dat-hai-phong) [livescore](http://m.ibongda.vn/livescore) [kết quả xổ số mega 6/45](http://www.xosomega.com/ket-qua/xo-so-mega-6-45/) [truc tiep xo so](http://wap.soicauxoso.com/) [dự đoán xổ số](http://soicauxoso.com/du-doan-xo-so) [thống kê xổ số](http://soicauxoso.com/thong-ke-xo-so) [Premier Residences Phú Quốc](http://lotuslandvn.com/premier-residences-phu-quoc/) [link sopcast](http://soccer188.net/link-sopcast) [premier league sopcast](http://soccer188.net/link-sopcast/english-premier-league.s1) [thiết bị định vị](http://thegioidinhvi.vn/thiet-bi-dinh-vi) [trực tiếp xổ số mega 6/45](http://www.xosomega.com/645/truc-tiep-ket-qua-xo-so-mega-645/) [thiết bị định vị ô tô](http://thegioidinhvi.vn/thiet-bi-dinh-vi/thiet-bi-dinh-vi-o-to) [trực tiếp xổ số](http://soicauxoso.com/xo-so-truc-tiep) [cuoi xuyen viet](http://youtube.mtube.vn/tag/cuoi+xuyen+viet+chung+ket+6) [giong hat viet 2015](http://youtube.mtube.vn/tag/giong+hat+viet+2015) [kết quả xổ số](http://soicauxoso.com/ket-qua-xo-so) [đại lý xổ số mega 6/45](http://www.xosomega.com/645/lien-he-lam-dai-ly-xo-so-tu-chon-vietlott/) [thach thuc danh hai](http://youtube.mtube.vn/tag/thach+thuc+danh+hai) [Cẩm nang Đầu Tư Premier Village Phu Quoc](http://lotuslandvn.com/cam-nang-dau-tu-du-an-premier-village-phu-quoc-resort/) [5s online](http://youtube.mtube.vn/tag/5s+online) [co dau 8 tuoi](http://youtube.mtube.vn/tag/co+dau+8+tuoi) [dang ky m88 nhu the nao](http://cacuoc247.com/) [live scores](http://m.soccer188.net/) [serie a sopcast](http://m.soccer188.net/sopcast)


markdown:
Toggle navigation
List Truyện
Ngẫu nhiên
Mới update chap
Truyện mới
Truyện tranh A-Z
Nhiều người xem
Đánh giá cao
Đang tiến hành
Tạm ngưng
Đã hoàn thành
Tất Cả
Action
Adult
Adventure
Anime
Bishounen
Comedy
Cooking
Demons
Doujinshi
Drama
Ecchi
Fantasy
Gender bender
Harem
Hentai
Historical
Horror
Josei
Live action
Magic
Manhua
Manhwa
Martial arts
Mature
Mecha
Medical
Military
Music
Mystery
One shot
Oneshot
Other
Psychological
Romance
School life
Sci fi
Seinen
Shotacon
Shoujo
Shoujo ai
Shoujoai
Shounen
Shounen ai
Shounenai
Slice of life
Smut
Sports
Super power
Superna
Supernatural
Tragedy
Vampire
Webtoons
Yaoi
Yuri
Chap Mới
Ngẫu Nhiên!
Đăng nhập
Chào mừng, iuTruyenTrannh Phiên bản mới. Nhanh hơn - Chuyên nghiệp hơn - Tiện hơn!
Tìm truyện Manga
Trang chủ
Tìm truyện nâng cao
Từ khóa:
Tìm
Thêm Tùy chọn
Reset
Tác giả/họa sĩ:
Thể loại:
Action
Adult
Adventure
Anime
Bishounen
Comedy
Cooking
Demons
Doujinshi
Drama
Ecchi
Fantasy
Gender bender
Harem
Hentai
Historical
Horror
Josei
Live action
Magic
Manhua
Manhwa
Martial arts
Mature
Mecha
Medical
Military
Music
Mystery
One shot
Oneshot
Other
Psychological
Romance
School life
Sci fi
Seinen
Shotacon
Shoujo
Shoujo ai
Shoujoai
Shounen
Shounen ai
Shounenai
Slice of life
Smut
Sports
Super power
Superna
Supernatural
Tragedy
Vampire
Webtoons
Yaoi
Yuri
Độ Tuổi:
Sao cũng được
13 +
14 +
15 +
16 +
17 +
18 +
Kết quả tìm

Server: LiteSpeed

Vary: User-Agent