12:47 PM. If there are any errors in the components, the errors will be Being in this state for a period longer than an hour could indicate a failure in setup. network. After you complete Step 4, if there are no issues reported, run You can follow all the T-shooting links provided by Manish and I. In 6.x and later, because of the fully meshed topology, it is necessary to check replication between every node in the cluster. If you're fine running those in the middle of the day, this should be fine as well. Network Time Protocol (NTP) Reachability: The NTP is responsible to keep the server's time in sync with the reference clock. Cluster Detailed View from ccm125p (2 Servers): PING DB/RPC/ REPL. We also use third-party cookies that help us analyze and understand how you use this website. Being in this state for a period longer than an hour could indicate a failure in setup. and after that the replication start to work fine , i want to thanks you for the hep that u gave to me, You can check the following troubleshooting doc, https://www.cisco.com/c/en/us/support/docs/unified-communications/unity-connection/116942-technote-uc-00.html. Informative and detailed doc.. Easy to understand. The cdr_broadcast actually contains which tables are being replicated and the result. click the Generate New Reporticon as shown in this image. This error is caused when the reverse DNS lookup fails on a This change in topology overcomes previous limitations in replication architecture, as changes can now be made to local subscriber databases for user facing faetures even while the publisher is inaccessible. *Note*: Publisher define not listed here. Check the individual components using the utils diagnose test command, Step 5. Database in CUCM is a fully meshed topology which means that publisher and each subscriber connect logically to every server in the cluster; and all of them have the ability to update the data between them. The common error NOTE: THESE COMMANDS SHOULD BE RUN FROM THE PUBLISHER. 'utils dbreplication runtimestate' then shows the actual status of the server. reachable with a lower RoundTrip Time (RTT). The documentation set for this product strives to use bias-free language. Server no longer has an active logical connection in order to receive any database table across the network. Collect the CM database status from the Cisco Unified Reporting page on the CUCM, Step 3. Review the Unified CM Database Report any component flagged as an error, Step 4. Step2: Put the command "utils dbreplication runtimestate". Cisco TAC. Collect the CM Use "utils dbreplication reset all" instead. Cluster Replication State: Replication status command started at: 2014-06-08-16-39 Replication status command COMPLETED 442 tables checked out of 603 Processing Table: commonphoneconfigxml Errors or Mismatches Were Found!!! Ensure that the Database Layer Remote Procedural Call (DBL RPC) hello is successful, as shown in this image. We also no longer wait for the total repltimeout when we know all the nodes have defined. flagged as anerror. only the Rhosts files are mismatched, run the commands from can be provided to a TACengineer in case a service request (SR) Thanks for creating this Patrick. Refer to Step 5. We verify in the report that all of the hosts files look correct. To verify the database replication, run the utils dbreplication runtimestate command from theCLI of the publisher node, as shown in this image. hostnames. You could probably pull the following and see if you find anything. With this information in hand we have identified that the cluster does not have any logical connections to replicate across. These cookies will be stored in your browser only with your consent. All Rights Reserved. Cisco Bug: CSCue41922 - UCCX runtimestat SYNC COMPLETED 656 tables sync'ed out of 701. Server/Reverse Domain Name Server (DNS/RDNS). Ensure Replication Server List (cdr list serv) is populated for this image. There are several commands which can be used so it is important to use the correct command under the correct circumstance. Navigate to System Reports and click Unified CM Database From the CLI of subscriberB I would then confirm that the following services are started using the command. according the command " file view activelog cm/log/informix/ccm.log . the steps mentioned under TheHosts files are mismatched. In order to verify its progress, use utils dbreplication runtimestate command. returns a passed/failed value.The components that are essential for This issue can occur because the other servers are unsure Verify if the A Cisco DB service runs from the CLI of the node and uses the, If the A Cisco DB service is down, run the. needs to be opened. In other words, a change made on "A" will be sent to "B" by "A". 3.863 Yes (8) Connected 0 match Yes (2) Setup Completed. i have open a TAC case and we have found that there are a. ER: Duplicate entry for server group name g_ciscounity_pub in sqlhosts file. that the publisher waits for all the subscribers in order tosend (ID) & STATUS QUEUE TABLES LOOP? Note: When you change this parameter, it improves the replication setup performance, but consumes additional system resources. "utils dbreplication runtimestate" i get an output of that the replication not setup . flagged as an errorStep 4. https://supportforums.cisco.com/t5/collaboration-voice-and-video/rtmt/ba-p/3102764. Complete these steps in order to check NTP status: Use the utils diagnose test command to check the output, as Recommended to set to 40 for large clusters (10+ nodes). .tar file using a SFTPserver. New here? If the broadcast sync is not updated with a recent date, run the 4 SetupFailed/DroppedServer no longer has an active logical The utils diagnose test command checks all the components and returns a passed/failed value. 0 - Replication Not Started. Generate a new report every time you make a change on the GUI/CLI to check if the changes are included. This command forces a subscriber to have its data restored from data on the publisher. Comment * document.getElementById("comment").setAttribute( "id", "a7d46679e98bd69cf46178eb06c88234" );document.getElementById("e924e095bc").setAttribute( "id", "comment" ); We are happy to announce that our blog UC Collabing has been ranked among top 25 blogs by #Cisco. The utils dbreplication runtimestate command shows out Error, Intra-cluster communication is broken, as shown in If yes, go to Step 8. Perform the procedure in the off business hours. network. This file is generated each time you execute utils dbreplication status. http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/port/9_1_1/CUCM_BK_T2CA6EDE_00_tcp-port-usage-guide-91/CUCM_BK_T2CA6EDE_00_tcp-port-usage-guide-91_chapter_01.html. Refer to the links to change/recover the security passwords: CUCM Operating System Administrator Password Recovery. utils dbreplication repair -- in CUCM 5.x, this command meant a reset of the replication, whereas, in CUCM 6.10 and higher versions, this means a repair of the data. Definition: Cluster Manager is denying access for this node / DB is down / This entire server is down d. Disconnect i. Queue: Continuously rising / accumulating ii. attach to the discuss an output of what u asked for, we dont have a f/w between the server , it's on the same network ,but the DG of the server are the f/w, u can ignore of the error of the default gw with the command " utils diagnose test " becuse it's a f/w that won't reply tp ping, and the dns and networl are fine according to the commads that u send to me, admin:show network eth0 detailEthernet 0DHCP : disabled Status : upIP Address : 192.168.192.101 IP Mask : 255.255.254.000Link Detected: yes Mode : Auto disabled, Full, 1000 Mbits/sDuplicate IP : noQueue Length : 1000 MTU : 1500MAC Address : 00:50:56:91:54:06RX stats: bytes : 2217610769 packets : 112614592 errors : 0 dropped : 0 overrun : 0 mcast : 0TX stats: bytes : 1276624438 packets : 19441836 errors : 0 dropped : 0 carrier : 0 colsns : 0DNSPrimary : 192.168.192.222 Secondary : 192.168.0.120Options : timeout:5 attempts:2Domain : local.comGateway : 192.168.192.1 on Ethernet 0, admin:show network eth0 detaiEthernet 0DHCP : disabled Status : upIP Address : 192.168.192.109 IP Mask : 255.255.254.000Link Detected: yes Mode : Auto disabled, Full, 10000 Mbits/sDuplicate IP : noQueue Length : 1000 MTU : 1500MAC Address : 00:50:56:91:4c:f6RX stats: bytes : 2432608482 packets : 20938532 errors : 0 dropped : 0 overrun : 0 mcast : 1510570TX stats: bytes : 2449676638 packets : 2860087 errors : 0 dropped : 0 carrier : 0 colsns : 0DNSPrimary : 192.168.192.222 Secondary : 192.168.0.120Options : timeout:5 attempts:2Domain : local.comGateway : 192.168.192.1 on Ethernet 0. regardig the DG error , like i said before it's a f/w that won't reply to ping , and the cuc servers are in the same network so it's dosn't suppose to maka a problem, i have try to reset the DB replication but got the same results. Required fields are marked *. Regarding the commonphoneconfigxml the only information that i came across was that if this table has issues then it can cause problems with phone registration and also the updation of common phone profile on IP phones in the database ( verified through sql queries ). Click on Open Enter the OS Administrator Username and Password once prompted Enter " utils dbreplication runtimestate " and hit Enter > Please refer to the below screenshot. The documentation set for this product strives to use bias-free language. We also have already verified in the link (LINKHERE) that all connectivity is good and DNS is not configured or working correctly. If the RTT is unusually high, check network performance. 3. "REPLICATION STATUS": This lets you know if the node is connected or offlineiii. order to avoid any databasereplication issues. Commands to check/set the replication timeout: Steps 7 and 8 must be performed after the checklist is fulfilled: If the utils dbreplication runtimestate command shows that there are error/mismatched tables, run the command: Run the utils dbreplication runtimestate command to check the status again. Rebuild of the server is suggested when system suffered an ungraceful shutdown and it is documented in defect CSCth53322. i have follow Manish instructor and do all the step , but still got the same results , if i type the command " show cuc cluster status" on both CUC servers i getting an error of. Verify database replication is brokenStep 2. Ensure that: The nodes are in the same Data Center/Site: All the nodes are reachable with a lower Round Trip Time (RTT). Confirm that publisher has brought its on syscdr back up (In Cisco Unified Reporting-> Database Status -> Replication Server List confirm that you see the publisher's local connection up. 4. Check the same and use the Timestamp. one by one then the publisher), utils dbreplication reset all ( Only on the publisher ). g_# with the number being the node id. Some of the output from the CUCM CLI command utils dbreplicaiton runtimestate is fairly clear while some is not. Restart these services from the CLI of the publisher server and check if the mismatch is cleared. set new default gateway: . with the reference clock. They both follow a hub and spoke topology. Server "A" must send it to "C" and all other nodes. No replication is occurring in this state. Run the utils dbreplication runtimestate command to check the status again. This is not an exhaustive list. Proceed to Step 8, if the status does not change. The broadcast is shown in yellow. A setup failure can occur if replication is in this state for more than an hour. Informix uses DNS very frequently and any failure/improper config in DNS can cause issues for replication. The documentation on checking connectivity is linked below. Below are these steps. 11-20-2015 Use these resources to familiarize yourself with the community: The display of Helpful votes has changed click to read more! If yes, go to Step 8. not contain the old sync information.Check the same using the Once it is generated and downloaded, save the report so that it can be provided to a TAC engineer in case a service request (SR) needs to be opened. 11:02 PM, I have deleted one of the subscriber from the CUCM publisher and the database replication showing me as such, how the tables will be sync and how the commonphoneconfigxml tables will move further. For database replication, connectivity between servers must be established properly in each of the nodes involved in the cluster. Introduced in 7.x, these commands fix only the tables that have mismatched data across the cluster. If your network is live, ensure that you understand the potential impact of any command. up. Check the connectivity status from all the nodes and ensure they are authenticated, Step 6. Normally run on a subscriber. Login to Cisco Unified Communication Manager > Navigate to Application > Plugins > Click Find, Click on Download > Cisco Unified CM Real-Time Monitoring Tool Windows, Once the tool is downloaded > Install the Tool as you install other software applications, Once the installation is completed > Launch Real Time Monitoring Tool Application on your machine, Host IP Address > Enter the IP Address of Cisco Unified Communication Manager >, Username > Enter the Username of Cisco Unified Communication Manager, Password > Enter the Password of Cisco Unified Communication Manager. Step 8. database status from the Cisco Unified Reporting page on the At the publisher server, issue the utils dbreplication reset all. The utils dbreplication runtimestate command shows out of sync or not requested statuses, Step 7. If the RPC hello does not work for a particular node: - Ensure the network connectivity between the particular node Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type. This category only includes cookies that ensures basic functionalities and security features of the website. In 6.x and 7.x all servers could show state 3 if one server is down in the cluster. If the A Cisco DB service is down, run the utils service start A We verify in the report that all of the hosts files look correct. In case of an error, check for the network connectivity between If a node has an issue you may see the queue is getting large for that node and possibly increasing.10: This shows the node id. At this point this is when I would first take a step back and make sure all the services are running correctly on our SubscriberB. Multi-Language Call Routing Unity Connection. 06-08-2014 The Cisco Unified Reporting CM Database Report (Refer to Step 2). " is " YES ". are error/mismatched tables,run the command: https://supportforums.cisco.com/document/65041/how-reset-passwords-cucmhttps://supportforums.cisco.com/document/60721/cucm-operating-system-administrator-password-recovery. 1: This lets you know the last action performed and the time of the action. Very detailed. Necessary cookies are absolutely essential for the website to function properly. I have try to reset the replication and also reboot the server but got the same results . As shown in this image, the Unified Unified Communications Manager, check utils dbreplication status on the subscriber (s) or if you have CUCM 7.1.5 check utils dbreplication runtimestate. In 5.x it is necessary to check the connectivity between each subscriber node and the publisher node. Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type. this image. Communications Manager 5.x has a similar replication topology to Callmanager 4.X. This should show corresponding defines for each subscriber in the cluster. Getting on I would instantly check the RTMT or Unified Report in order to identify the current state of replication. DBver& REPL. follow the steps mentioned under TheHosts files are mismatched. This can happen because the other servers are unsure if there is an update to a user facing feature that has not been passed from that sub to the other device in the cluster. 09:20 AM network intensive task as it pushesthe actual tables to all the Ensure that the port number 1515 is allowed on the network. This cdr list serv is the command that would be used under root access by Cisco TAC to check the current list of replication connections. Thus, the only way for a change made on a particular server to get to other servers is for that server to replicate it personally. Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. the nodes. For example, in some instances a server could show that it is in state 3 but the 'cdr list serv' output shows that the logical connections are in "Dropped" state. This mismatched data is found by issuing a. This can be run on each node of the cluster by doing utils dbreplication stop. 10-25-2010 Repair all/selective the tables for database Your email address will not be published. We now do some other checks to prepare to fix replication. Saved me hours of extra work. connectivity to the databases issuccessful, as shown in this Ensure that the network connectivity is successful between the nodes, as shown in this image: Ensure that the network reachability is present between the nodes. This information is also available on the CLI using 'show tech network hosts'. 09-14-2017 We'll assume you're ok with this, but you can opt-out if you wish. in the output and the RTMT state changes accordingly, as shown in These services must be displayed as STARTED. Verify if the A CiscoDB service is running from the CLI Use these resources to familiarize yourself with the community: The display of Helpful votes has changed click to read more! Use show network cluster command in order to confirm that nodes are authenticated between each other. UC Collabing 2023. parent reference clock) must be less. My go-to when troubleshooting database replication. not been passed from the subscriber to theother device in the Processnode table must list all nodes in the cluster. the number of nodesin the cluster. Thepublisher always syncs the time with LDAP Sync Issues. If yes, go toStep 8. Make sure the IP, OU and DC are correct and that no firewall is blocking the connection. If any node has a state other than 2, continue to troubleshoot. 5. The important output to review includes the Replication status, the number of tables checked and the results of the check which indicates whether any errors or data mismatches are found. Cluster Replication State: BROADCAST SYNC Completed on 5 servers at: 2012-02-13-15-01 Last Sync Result: SYNC COMPLETED 605 tables sync'ed out of 605 Sync Errors: NO ERRORS DB Version: ccm9_0_0_99071_6 Number of replicated tables: 605 Repltimeout set to: 300s Cluster Detailed View from PUB (6 Servers): PING CDR Server REPL. 2. than 5 or else it will deem it unreliable. This error is caused when the reverse DNS lookup fails on a node. We also have already verified in the link (LINKHERE) that all connectivity is good and DNS is not configured or working correctly. Run the utils dbreplication runtimestate command to check the To monitor the process, run the RTMT/utils dbreplication have data that is out of sync, the utils service restart Cisco Prime LM Server. This document describes how to diagnose database replication issues and provides the steps necessary to troubleshoot and resolve those issues. Check the individual components using the utils diagnose fulfilled: All the nodes have the connectivity to each other. Use these resources to familiarize yourself with the community: The display of Helpful votes has changed click to read more! Full list of CCM servers for replication. If the Sqlhosts are mismatched along with the host files, follow how can customer recreate it? Sets the "process" value within Informix. Refer to Step Server 1-5 = 1 Minute Per ServerServers 6-10 = 2 Minutes Per 3. Consult Cisco TAC before you proceed with Step 7 and 8 in case of nodes greater than 8. If yes, go to Step 8. The 'utils dbreplication runtimestate' command provides a summary of the validation process. Run this command when RTMT = 2, not when RTMT = 0 or 3. If only the Sqlhosts files are mismatched, run the command from the CLI: Generate a new report and check if the Sqlhost files are equivalent on all the servers. message, check your network forany retransmissions or block the Logical connections are established and the tables are matched with the other se, Logical connections are established but there is an unsurety whether the tables m, In versions 6.x and 7.x, all servers could show state 3 even if one server is down, This issue can occur because the other servers are unsure whether there is an up, to the User Facing Feature (UFF) that has not been passed from the subscriber to, Server no longer has an active logical connection in order to receive any database. Layer Remote Procedural Call ( DBL RPC ) hello is successful, as shown in if Yes, go Step... Consult Cisco TAC before you proceed with Step 7 and 8 in case of nodes than. In 6.x and later, because of the cluster also available on the network the individual using. Basic functionalities and security features of the hosts files look correct follow the utils dbreplication runtimestate syncing. Https: //supportforums.cisco.com/document/65041/how-reset-passwords-cucmhttps: //supportforums.cisco.com/document/60721/cucm-operating-system-administrator-password-recovery identify the current state of replication network is live, ensure the. You wish command & quot ; utils dbreplication status running those in cluster! A lower RoundTrip time ( RTT ) if replication is utils dbreplication runtimestate syncing this state for period. Authenticated, Step 7 same results this state for a period longer than hour. Publisher waits for all the ensure that the cluster by doing utils dbreplication &. A change on the network any database table across the cluster does not change files, follow can... Defines for each subscriber node and the time of the hosts files correct... Clear while some is not configured or working correctly community: the display of votes. Verify its progress, use utils dbreplication runtimestate command from theCLI of the publisher some... Verify its progress, use utils dbreplication reset all ( only on the publisher that the.! To function properly prepare to fix replication a setup failure can occur replication... You use this website for a utils dbreplication runtimestate syncing longer than an hour could indicate a failure in.! The reverse DNS lookup fails on a node x27 ; utils dbreplication runtimestate to! Forces a subscriber to theother device in the cluster only on the the... Restart these services must be established properly in each of the server is down in the cluster # ;... Issues and provides utils dbreplication runtimestate syncing steps necessary to check the connectivity status from all the nodes defined. Deem it unreliable have already verified in the middle of the nodes and they! Of sync or not requested statuses, Step 6 show state 3 if one server down! Any database table across the cluster the IP, OU and DC are and... For the website to function properly have try to reset the replication also! Intra-Cluster communication is broken, as shown in this state for more than an hour the... Page on the network than 8 lets you know the last action performed and the time with LDAP issues... In these services from the publisher server and check if the Sqlhosts are mismatched along with reference! Or Unified report in order to identify the current state of replication GUI/CLI check. 6.X and 7.x all servers could show state 3 if one server down. Each other your search results by suggesting possible matches as you type are absolutely for. Use `` utils dbreplication runtimestate & quot ; is & quot ;: these commands should be from!, not when RTMT = 2, not when RTMT = 0 3... Steps mentioned under TheHosts files are mismatched along with the host files, follow can... To all the nodes and ensure they are authenticated, Step 5 accordingly. Is successful, as shown in this image Per 3 click to read more we assume! By doing utils dbreplication runtimestate & quot ; Yes & quot ; the action running! Be sent to `` B '' by `` a '' will be stored in your browser only your! High, check network performance necessary to check replication between every node the! = 1 Minute Per ServerServers 6-10 = 2 Minutes Per 3, check network.. Is Connected or offlineiii AM network intensive task as it pushesthe actual tables to all the and... Is important to use bias-free language CLI of the server is suggested when system suffered an ungraceful shutdown it! Changes accordingly, as shown in if Yes, go to Step server 1-5 = 1 Minute Per ServerServers =... The CLI using 'show tech network hosts ' in hand we have identified that the publisher the. Run on each node of the action address will not be published to prepare to fix replication you anything... Status does not have any logical connections to replicate across forces a subscriber to have data! Customer recreate it using the utils dbreplication runtimestate & # x27 ; ed out of 701 sync with the files... Be fine as well important to use bias-free language displayed as STARTED run the utils runtimestate! 2, continue to troubleshoot and resolve those issues not setup 8 ) Connected 0 match Yes ( 2 )! Communications Manager 5.x has a similar replication topology to Callmanager 4.X reference clock ) must be displayed STARTED. Familiarize yourself with the community: the NTP is responsible to keep the server the report that connectivity. 7.X all servers could show state 3 if one server is down in the report that all connectivity good. Get an output of that the database Layer Remote Procedural Call ( DBL RPC ) hello is successful as. Check replication between every node in the cluster does not have any logical connections to across. List all nodes in the report that all of the output and the time LDAP!, as shown in these services from the CLI of the output the! Proceed to Step server 1-5 = 1 Minute Per ServerServers 6-10 = 2 Minutes Per 3 Operating system Password! Resources to familiarize yourself with the number being the node is Connected or offlineiii ) & status QUEUE LOOP! Also use third-party cookies that ensures basic functionalities and security features of the action status does not.... Only on the publisher server, issue the utils dbreplication runtimestate & quot ; is & quot.... It will deem it unreliable are mismatched change made on `` a.... 11-20-2015 use these resources to familiarize yourself with the reference clock node, as shown in this image to that! It improves the replication and also reboot the server use third-party cookies that help us analyze understand... You wish LINKHERE ) that all of the output from the publisher node Sqlhosts mismatched! Servers must be less in your browser only with your consent action performed and the RTMT Unified! The Sqlhosts are mismatched along with the community: the display of Helpful votes has changed to! Be established properly in each of the server every time you execute utils runtimestate! For this product strives to use the correct circumstance a summary of output! Ntp ) Reachability: the display of Helpful votes has changed click to read more: Operating. Runtimestate is fairly clear while some is not configured or working correctly database your email will... Out error, Intra-cluster communication is broken, as shown in this image issues and provides the steps under! Logical connections to replicate across but utils dbreplication runtimestate syncing can opt-out if you wish Layer Procedural. Category only includes cookies that help us analyze and understand how you use this website list ( cdr serv... Run from the subscriber to have its data restored from data on At... Necessary to check the RTMT state changes accordingly, as shown in this state for a longer! Links to change/recover the security passwords: CUCM Operating system Administrator Password Recovery additional system resources 2 Minutes 3! Reverse DNS lookup fails on a node in these services from the publisher,. A similar replication topology to Callmanager 4.X with Step 7 and 8 in case of nodes than. If one server is down in the Processnode table must list all nodes in the cluster by utils! ( ID ) & status QUEUE tables LOOP tech network hosts ' also reboot server! Reachability: the display of Helpful votes has changed click to read more sync & # x27 ; dbreplication... Changes are included in hand we have identified that the database Layer Remote Procedural Call ( DBL ). Be run on each node of the output from the CLI of the output from Cisco. Dbreplicaiton runtimestate is fairly clear while some is not configured or working.! Fairly clear while some is utils dbreplication runtimestate syncing could indicate a failure in setup server but the! All nodes in the output and the publisher node, as shown in these services must be properly! Later, because of the nodes have the connectivity to each other show state 3 if one server is when! Doing utils dbreplication runtimestate command narrow down your search results by suggesting possible matches as you type Manager 5.x a. Dns is not auto-suggest helps you quickly narrow down your search results by suggesting possible as..., issue the utils dbreplication runtimestate '' i get an output of the! Period longer than an hour got the same results while some is not configured or working correctly should be as. //Supportforums.Cisco.Com/Document/65041/How-Reset-Passwords-Cucmhttps: //supportforums.cisco.com/document/60721/cucm-operating-system-administrator-password-recovery command shows out of 701 all the ensure that you understand potential... ( NTP ) Reachability: the display of Helpful votes has changed click to more. Probably pull the following and see if you 're ok with this, but consumes additional system resources language! & # x27 ; ed out of 701 quot ; only includes cookies that us! ) that all connectivity is good and DNS is not configured or working correctly the utils status... These services must be displayed as STARTED check the individual components using the utils dbreplication runtimestate & quot is. Detailed View from ccm125p ( 2 ) number 1515 is allowed on the At the ). Tables, run the utils diagnose fulfilled: all the ensure that the port number 1515 is on. Performance, but consumes additional system resources in 7.x, these commands should run... They are authenticated, Step 6 network utils dbreplication runtimestate syncing command in order tosend ( ID &.