- Welcome BMC Customer Support Site Guest User
Knowledge Article
Remedy - FTS - I have multiple FTS Indexers in my environment, can I use all of them for searching? - INCLUDES VIDEO
I have multiple FTS Indexers in my environment, can I use all of them for searching? - INCLUDES VIDEO
Remedy AR System Server
AR System
ARServer 9.x FTS
I have multiple (3) FTS Indexers in my environment. I setup these up for High Availability. The documentation states that all the user facing servers should point to the #1 Ranked FTS Indexer. So all of my FTS and Global Search queries are being funneled through a single FTS Searcher Plugin. Is it possible to distribute the FTS Search load across all of my Indexers? |
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers. Note: For the complete FTS Deployment and Configuration Whitepaper, see KA000159890 For the ITSM Configuration Best Practices document, please see KA000114508
For the FTS Best Practices Only (not the whitepaper) see KA000153129
Answer: Yes, you can. Note: Before we get started, there is one downside to doing this. If your Indexers are not all in sync, your end users could get different results depending on which AR Server they connect to. In order to minimize this possibility, check the contents of the ft_pending table in the database and make sure that there are similar counts for each of your indexer servers. You can run the following SQL Query to get an idea: select count(*), indexservername from ft_pending group by indexservername; The count will show how many entries have yet to be indexed for each server. If your servers have similar counts, there are pretty much in sync. If they are not very similar, you might want to look into that before using this method. See "What to do if your indexes become out of sync" below for more information. Assuming that your indexer servers are working properly and that your User Facing servers are connecting fine and are able to search properly, this is a fairly simple process. Using a Load Balancer 1- The first step is to decide if you want to use a Load Balancer in front of the FTS Plugin Servers or not. If you want your load distributed evenly from all of your user facing servers, you can put a Load Balancer between your ARServers and your FTS Searcher Plugin Servers. a. In order to configure for this, you will have to configure the load balancer to direct traffic to the FTS Plugin Servers using the correct plugin port (9977 by default). You can also configure a failover group so that if a plugin server is unresponsive, it automatically routes to a different plugin server. b. You will need to set each User Facing server so that FTS points to the Load Balancer instead of the actually FTS Plugin Servers. This can be done by changing the value of the 'Indexer Server Name' on all User Facing servers to a name that resolves to the Load Balancer. c. You will need to restart the User Facing servers for this setting to take effect. Once this is done, all FTS and Global Searches should be distributed evenly across the group of FTS Ranked Servers. Note: Since you are adding a network device that can failover and FTS already has failover capabilities, it is important to understand how they will work together. 1. If one of the ARServers goes down, the co-related FTS Plugin server will also go down.
In this case, the Load Balancer will not forward requests to that server but will send the request to another FTS Plugin Server that is still available. 2. If one of the FTS Plugin Servers has terminated or is in a state that it cannot respond, the Load Balancer may still forward a search request to it. In this case, the AR Server will re-reroute the request based on the FTS Ranking. It will try to send the request to the #1 Ranked FTS Server. If this happens to be the server that is down (or is down for some other reason), AR Server will then re-route the request to the #2 Rank FTS Server and so on. 3. If the Load Balancer happens to route the search request to an FTS Plugin that is currently busy performing a reindex, the ARServer will re-route the request based on the FTS Ranking. 4. Each API call goes through process so just because 1 request went to an unavailable server does not mean that the next one will. Each will follow the process just mentioned. Using Manual Load Balancing (no Load Balancer) 2. If you want to distribute FTS load but do not want to use a Load Balancer, you can manually configure your environment. a. You will need to set each User Facing server so that FTS points to a specified FTS Plugin Server.
This can be done by changing the value of the 'Indexer Server Name' of each User Facing servers to an individual FTS Server's plugin server. a. Since your User Facing servers were initially configured to connect to the #1 Ranked FTS server, you only need to config User Facing servers that will be pointing to one of the other FTS Servers. You will need to restart any AR Servers that you changed. You have now distributed your FTS and Global Search search requests across your FTS Indexer Servers. Note: Since we are bypassing the default configuration, it is important to understand how failover will occur. 1. If one of the FTS Plugin Servers has terminated or is in a state that it cannot respond, the User Facing AR Server will re-reroute the request based on the FTS Ranking. It will try to send the request to the #1 Ranked FTS Server. If this happens to be the server that is down (or is down for some other reason), AR Server will then re-route the request to the #2 Rank FTS Server and so on. 2. If the User Facing server is configured to route the search request to an FTS Plugin that is currently busy performing a reindex, the ARServer will re- route the request based on the FTS Ranking. 3. Each API call goes through process so just because 1 request went to an unavailable server does not mean that the next one will. Each will follow the process just mentioned. Note: What to do if your indexes become out of sync If you have an Indexer server go down for some period of time, such as for performing routine maintenance, you may have a situation in which the ft_pending jobs for a single server are piling up. When you bring that server back up, it will take time to catch up with the queue. But because that server has become available, searches will still be performed, giving inconsistent results to end users. It is best to prevent users from searching on this indexer until it is caught up. As mentioned earlier, you can run the following SQL to check how out of sync your indexers have become: select count(*), indexservername from ft_pending group by indexservername; Once you bring the server back up, immediately stop the fts searcher plugin server. This can be done easily in v.18.05, v.18.08 and v.9.1.04 where the Server Group Administration Console has been installed. For v.9.1.04, this is an option available from https://communities.bmc.com/docs/DOC-102081 a) From the Server Group Administration Console, open the "Manage Processes" page. b) Sort by Process Name and look for the BMC:FtsJavaPluginServer entry. c) Identify the exact line for the server that is out of Sync. d) Highlight that line and choose an Operation of "Stop". e) Click on the Execute button. This will stop the plugin server and searches will not be performed on this Indexer server. f) Periodically run the SQL until your indexers are in-sync. g) Then highlight the stopped BMC:FtsJavaPluginServer and choose Operation 'Start' and execute. Searches will now be directed to this indexer server. |
Attachment(s):