With thanks to CBC Poland,  Cortrol.eu

Failover beveiligt uw systeem

Een degelijk video management systeem kan niet bestaan zonder mechanismen die de bedrijfszekerheid garanderen.
CORTROL Global biedt verschillende mogelijkheden die de continuïteitsgraad verhogen.

Database backup, Management server mirroring, Failover, Fallback, Database replicatie, Edge recording.

Gecombineerd herleiden die de onderbreking van de video streaming tot nul, met behoud en voortzetting van de integrale video opname.

In dit artikel bespreken we het Failover systeem in CORTROL Global en bekijken we de instellingen die de redundantie van het video management system verhogen.

Wat is Failover

De Failover functie maakt een snelle en geautomatiseerde herstel van het opnamesysteem mogelijk in geval van een plotse storing.
CORTROL Global, geïnstalleerd op de Failover Server, volgt de werking op van alle servers in het camerasysteem, door het uitwisselen van hartslagberichten (hartbeats)
Als de hartslag wegvalt en een respons laat te lang op zich wachten, dan gaat de Failover Server de uitgevallen Recording Server vervangen door een Spare Server.

De Failover Server beheert de configuratie van alle Recording Servers in het systeem.
De Spare Server ontvangt een copie van de instellingen van de uitgevallen Recording Server en neemt haar activiteit over (Live Streamen & Beeldopname).

Alle Recording Servers worden in groepen verdeeld (clusters). Elke groep of cluster van servers wordt een reserve server toegewezen de Failover Node.


CORTROL1failover

Er zijn enkele vereisten voor de Failover Node (Spare Server of Reserve Server).

De Failover Node bevindt zich in dezelfde omgeving als Recording Servers in de cluster.
De Failover Node en alle Recording Servers vinden elkaar, en elkaars IP-Camera's, in eenzelfde netwerk.
De hardware specificaties vande Failover Node is minstens identiek of beter dan de Recording Servers.
De Failover Node is niet geconfigureerd voor CORTROL video management software.
De CORTROL configuratie van de Recording Servers is niet permanent en wordt beheerd door CORTROL Global op de Failover Server.

Wat als de Failover Server uitvalt ?

Een Failover systeem bevat minimum 3 servers. De Failover Server beheert een Primary Recording Server en een Reserve Server.
Als de Failover Server uitvalt en de Recording Server niet, blijft het camerasysteem toch verder werken.
Server Mirroring is de oplossing continuïteit van een Failover Server te verzekeren.

*** Redundancy. Redundantie : overbodigheid. Bedoeld wordt echter : wisselstukken en reserve-onderdelen.

Set Up Once and for All

Failover is configured via CORTROL Console application. Generally, the plan is:

  1. Create a failover cluster
  2. Add all necessary recording servers to the CORTROL Global configuration — manually or using server autodiscovery
  3. Assign server roles (primary recorder or failover node) and define their settings
  4. Put the servers into the cluster(s)

The order of these steps is not crucial, e.g., you can first add the servers and create the failover cluster later.

We shall consider a system with one cluster that contains two primary recording servers and two spare servers. All these servers (four of them) are located in the same local network and, as a result, have access to the same cameras. This network can be different from the Global server network.

In Configuration / Servers, verzamel alle servers. Benoem een server tot "Failover Server" en andere server tot "Reserve Server". Benoem alle overige servers "Opname Servers X".

CORTROL2failover server list

 

Instellen van de Recording Servers

Double-click a recording server to bring up its properties editing dialog box, and switch to the Failovertab (do not forget to fill in the rest of the tabs afterward). Here, it is necessary to choose the failover cluster that servers will belong to, and define failover settings for this particular server.

Click Change to choose a pre-created failover cluster from the list. Do not worry if you have not created a cluster beforehand, you can do so right now by clicking the button in the bottom of the cluster list.

Dubbelklik op een Recording Server. Klik op Failover. Voeg de Recording Server toe aan een Failover Cluster (in het voorbeeld MostImportantClusterEver).

Current Failover Server blijft "none" en wordt straks door de Reserve Server toegewezen.

CORTROL3failover rec server settings

In the Current failover server field, it should be saying none at this point: here, the currently active failover node is shown, so this field will become non-empty once the target recording server fails and is replaced by a failover server.

The rest of the settings explained:

  • Failover timeout: for how long the Global server will wait after the last heartbeat message before it marks the target server faulty and replaces it with a failover node. We shall set this to the minimum equal to ten seconds;
  • Central server connection timeout: defines for how long the remote server should try receiving configuration from the Global server before giving up and starting with last good known configuration. Also set to minimum, one minute;
  • Auto recovery: if enabled. the remote recording server will start operating automatically once it is back to life. If the central server connection is available at that point, the failover server will be stopped by the Global server. We need to enable this in order to ensure autonomous system operation (otherwise, the failover will continue operating until manually replaced by the primary server);
  • Recovery timeout: delay before Global server activates the target server once it is back online; we shall set this to zero to make the recovered server resume operation without delay.

Repeat this for every primary recording server in the list.

 

Instellen van de Reserve Server.

For failover servers, the settings differ a bit. Double-click a server that is intended to be a failover node and stay on the Details tab: here, enable the Failover node role.

Dubbelklik op een Reserve Server.

Klik op Details tab. Vink aan Failover Node.

CORTROL4failover enable failover 

Vervolgens

Klik op Failover tab. Voeg de Reserver Server toe aan een Failover Cluster (in het voorbeeld MostImportantClusterEver)

CORTROL5failover failover settings 


Track Server Status

Current server status and also hardware load (for connected machines) can be viewed in the Monitoring section of CORTROL Console, under Servers.

  • If a recording server is offline and its duties are performed by a failover node, failover server status will be Substituted and its failover configuration will display the primary recording server name (whose configuration is currently used).
  • At the same time, the faulty recording server will have an Unknown status and will be marked red as unavailable.

Also, for each primary recording server, its current failover substitute is displayed in the server settings, Failover tab (as shown in the snapshot above).

Intended Effect

Once you have clustered the servers and configured them as described above, CORTROL Global is ready for the recording server misbehavior: whenever any of the two recording servers fails, it will be replaced by a failover server right away.
In addition to the camera configuration, the failover server sustains the state of the server event & action configuration.

From the point of view of the connected clients — CORTROL Monitor, mobile apps, and others — all failover operations are transparent for the user’s convenience.
Clients receive the requested live streams and recordings, and present these to their users, so the latter may not even suspect something might have gone wrong with any of the servers.

Recordings that have been made on failover servers remain there until they are erased, having reached one of the quotas. Individual archive duration quotas set for certain channels will affect all servers, i.e., the outdated recordings will be erased from the failover servers as well (provided, of course, that these servers are online and connected to the Global server).

Tips and Tricks

There are a few hints that can make your experience with CORTROL Global failover even more exciting ?

  • You can force a recording server to be replaced by a failover node before taking it down for maintenance, in this way eliminating the downtime (those several seconds necessary to re-initialize the streams). In this case, do not forget to set the recording server’s recovery timeout to be greater than zero so that you have time to turn it off! You can do this via Console > Configuration > Failover clusters > view cluster servers > Change failover server.
  • Server role (failover/primary) can be assigned already at the recording server auto discovery step.
  • No matter how fancy the recording servers’ storage configuration is (different labels for per-disk channel grouping etc), failover servers can simply have one capacious storage with a Defaultlabel.

Now to you are ready to set this up for your own CORTROL Global system. May your servers never break down!