Automatic failover... help

Hi there,

We've recently set up a Principle, Mirror and Witness configuration with the Mirror and Witness in a separate building to the Principle. All three are part of the same domain (DMZ) and are different servers, the buildings are connected via a fiber optic cable. All servers and SQL Server instances are logged in with the same domain admin account DMZ\esAdmin.

Mirroring is all set-up and the databases are synchronized. Every once in a while some (not all, normally 6 out of 15) databases will switch roles and become active on the mirror. The SQL Server mirroring monitor job then reports:

Date 25/01/2007 12:37:01
Log Job History (Database Mirroring Monitor Job)

Step ID 1
Server DMZSQL01
Job Name Database Mirroring Monitor Job
Step Name
Duration 00:00:02
Sql Severity 16
Sql Message ID 32038
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0

Message
Executed as user: DMZ\esadmin. An internal error has occurred in the database mirroring monitor. [SQLSTATE 42000] (Error 32038). The step failed.

I have no idea, what causes the failover, it could be a slow network or a bad set-up, can anyone give me some ideas of what to do to track down the problem or any experience of what could be causing this, it happens randomly every day or three. No warning and if I go to the mirror and failover back to the principle again then it's all just fine. However I don't want half my databases working on 1 server and half on the other.

Any ideas?

Thanks
Ed

UPDATE:

I've just been looking at the logs on my Mirror and at the same time it reports in this order

Error: 1479, Severity: 16, State: 1.

The mirroring connection to "TCP://DMZSQL01.dmz.local:5022" has timed out for database "WARCMedia" after 10 seconds without a response. Check the service and network connections.

Database mirroring is inactive for database 'WARCMedia'. This is an informational message only. No user action is required.

Recovery is writing a checkpoint in database 'WARCMedia' (41). This is an informational message only. No user action is required.

The mirrored database "WARCMedia" is changing roles from "PRINCIPAL" to "MIRROR" due to Failover.

Database mirroring is inactive for database 'WARCMedia'. This is an informational message only. No user action is required.

...

This looks like a time out, is there any way to set the TimeOut threashold for Database mirroring or set retry intervals?

Have you implemented network monitoring? I suggest you compare the failover times to the network monitoring history.

-Matt

|||

No, but any pointes on what to monitor would be usefull.

Thanks

Ed

|||

Couple of things to consider:

1. run with the safety OFF (then you wouldn't need the witness server). This won't allow automatic failover from with a real failure or a false failure. The false failures are what you seem to be seeing now, but it will allow your system to run in a more predictable manner, while you understand what is going on.

2. Consider moving the Witness (if you MUST run with the witness) to the building with the Principal. All things being equal, (i.e. if the probability of both building having a disaster is equal); then having the Witness and Principal share a "more reliable" connection should provide better reliability. Assuming: 1. servers closer together physically is more reliable. 2. You can't put the Witness in a third, equally reliable data center (who has that kind of $$$) 3. Your business needs dictate that you can put the Witness with the Principal

3. Increase the timeout "ping" for mirroring. ALTER DATABASE db SET PARTNER TIMEOUT = xx. (see BOL) By default, it is 10 seconds.

|||

Thank you all for your help.

I modified the timout pint to 60 seconds and I've not had any problems since. I have a feeling either there is something on the network causing a delay or the server was just under too much load at one off times. This has solved 1 problem so I'll continue to look into the cause over the coming weeks.

Thanks
Ed

Automatic failover with SQL Server Standard?

Hi,
We are trying to implement SQL server failover by using
Windows 2003 "Enterprise" clustering and SQL
Server "Standard" to try to save the cost of SQL Server
Etnerprise. From what I understand we cannot have
automatic failover of the SQL server service with this
configuration.
Could someone please explaing to me techinically why we
cannot do this:
- Implement W2003 Enterprise OS clustering.
- Install SQL server "standard" installed on both machines
in the OS cluster.
- Have the SQL server service start on the primary node in
the OS cluster and manage the database. Then ...
as part of the OS cluster confifuration, if the primary
node goes down, request that the SQL server service starts
up on the backup node as part of the OS failover.
Can't the clients continue to talk to the SQL server
database since it is not started up on the backup node and
has the same IP address?
Thanks!
As you rightly said, SQL Server 2000 Failover Clustering is NOT supported with SQL Server 2000 Standard Edition. It is fully supported on SQL Server 2000 Enterprise Edition.
You can install and configure any non-cluster aware application as a generic service on a cluster. You can even make notepad clustered as a generic service in a cluster if you like.
I have not seen anybody doing following this approach (using sql server 2000 std edition as a generic service). I wouldn't recommend it. This will be like a custom solution and if you have any issues with the failover
working properly, the support that you will get will be very limited. If you have to work with SQL Server 2000 Standard Edition then I would recommend exploring SQL Server Replication.
To get an idea of what SQL Server 2000 EE Failover Clustering does please refer to the following whitepaper.
Microsoft Whitepaper - SQL Server 2000 Failover Clustering
http://www.microsoft.com/technet/pro.../failclus.mspx
Cluster Support Policy
Microsoft's Cluster support policy is below
Cluster (General): http://support.microsoft.com/default...b;en-us;309395
Cluster (SQL): http://support.microsoft.com/default...b;en-us;327518
Best Regards,
Uttam Parui
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.
Are you secure? For information about the Strategic Technology Protection Program and to order your FREE Security Tool Kit, please visit http://www.microsoft.com/security.
Microsoft highly recommends that users with Internet access update their Microsoft software to better protect against viruses and security vulnerabilities. The easiest way to do this is to visit the following websites:
http://www.microsoft.com/protect
http://www.microsoft.com/security/guidance/default.mspx
|||Note that Veritas support clustering SQL2000 Standard. When I've used it,
it's been just using VCS and not involving MCS at all but I believe it can
sit on top if it as well. I think they've rebranded recently so its called
Storage Foundation High Availability rather than Veritas Cluster Server to
reflect its reliance on Veritas Volume Manager (which is now called Storage
Foundation). I actually prefer the Veritas cluster management tools to the
MS tools and it's a lot cheaper than going to Enterprise Edition if the only
reason you need Enterprise is clustering (and not performance/large memory
support etc).
HTH
Jasper Smith (SQL Server MVP)
http://www.sqldbatips.com
I support PASS - the definitive, global
community for SQL Server professionals -
http://www.sqlpass.org
<anonymous@.discussions.microsoft.com> wrote in message
news:2ec3601c46d96$f335fc60$a601280a@.phx.gbl...
> Hi,
> We are trying to implement SQL server failover by using
> Windows 2003 "Enterprise" clustering and SQL
> Server "Standard" to try to save the cost of SQL Server
> Etnerprise. From what I understand we cannot have
> automatic failover of the SQL server service with this
> configuration.
> Could someone please explaing to me techinically why we
> cannot do this:
> - Implement W2003 Enterprise OS clustering.
> - Install SQL server "standard" installed on both machines
> in the OS cluster.
> - Have the SQL server service start on the primary node in
> the OS cluster and manage the database. Then ...
> as part of the OS cluster confifuration, if the primary
> node goes down, request that the SQL server service starts
> up on the backup node as part of the OS failover.
> Can't the clients continue to talk to the SQL server
> database since it is not started up on the backup node and
> has the same IP address?
> Thanks!
>

Automatic Failover Problem

Setup Configuration:

3 servers
- PRINCIPAL IP: 10.2.5.31 - DNS Lookup: db-server-2.mosside.choruscall.com
- MIRROR IP: 10.2.5.30 - DNS Lookup: sql-mirror.mosside.choruscall.com
- WITNESS ip: 10.2.5.32 - DNS Lookup: sql-witness.mosside.choruscall.com

Each Server is running Windows Server 2003 Enterprise Edition with SQL Server 2005 Enterprise Edition.
All server instances are enabled for remote connections(By default they are not).
All servers have the flag 1400 traceon and have been restarted.
PORT 5022 is unrestricted on network.

The server instances are connecting via certificates. Each server has an endpoint for the certificates to to connect on.

Certificate Setup Proceedure:

Principal_Host:
1. Create Master Key with Password

2. Create certificate with subject

3. Create endpoint for certificate (Listener_Port = 5022, Listener_ip = all)
to connect on for database_mirroring

4. Backup Certificate (principal_cert.cer)

5. Take backed up certificate to Mirror_Host

(Reapeat Steps 1-5 for Witness and Mirror)

Mirror_Host: Create Certificate on Mirror_Host for inbound connections from Principal:

6.(On Mirror_Host) Create Login for Principal using same password in step 1 (principal_login)

7. Create user for login just created. (principal_user)

8. Create local certificate for Principal on Mirror using certificate generated by principal.

ex: Create Certificate Principal_cert Authorization Principal_user FROM FILE='c:\principal_cert.cer'

9. (If an endpoint has been created already on the mirror)Grant connectiion to the login:

ex: Grant connect on endpoint::mirror_endpoint to principal_login

Repeat Steaps 6-9 for Principal and Witness Servers accordingly.

10. Import Database to SQL Server 2005 Principal Instance

11. Backup Database to disk with format

12. Backup Database log file to disk with format

13. Copy backups to mirror

14. Restore Database and log file with norecovery on Mirror_Host

15. Configre Database for Database Mirroring on Principal Server
There are two ways to do this. Via the wizzard or via the Transact-SQL window.
Using the wizzard appears to work since I started using FQDN.

PROBLEM:

After configuration, everythig appears to be correct. That is, the principal displays
that it is the principal and it is synchronized with the mirror. The mirror also displays that it is the
mirror and it is synchronized with the principal and it is in recovery. If I failover manually, the mirror
becomes the principal and the principal becomes the mirror (They form a quarum). If I disconnect the principal
from the network, the mirror is supposed to form a quarum with the witness and promote itself to principal status.
This is not what is happening. The witness recognizes that the principal is down and logs that info into its log file.
The Mirror attempts to contact the witness but cannot log onto the machine. The Mirror Logs the following:

Error: 1438, Severity: 16, State: 2.
The server instance Witness rejected configure request; read its error log file for more information.
The reason 1451, and state 3, can be of use for diagnostics by Microsoft.
This is a transient error hence retrying the request is likely to succeed.
Correct the cause if any and retry.

<<<<<<<MIRROR SERVER >>>>>>>>

2007-09-06 15:08:45.32 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:08:45.32 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:05.32 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:05.32 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:25.33 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:25.33 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:45.34 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:45.34 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:10:05.35 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:10:05.35 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:10:25.36 spid23s Error: 1438, Severity: 16, State: 2.

<<<<<<< WITNESS SERVER >>>>>>>>

2007-09-06 14:19:55.90 spid52 The Database Mirroring protocol transport is now listening for connections.
2007-09-06 15:07:11.64 spid24s Error: 1479, Severity: 16, State: 1.
2007-09-06 15:07:11.64 spid24s The mirroring connection to "TCP://db-server-2:5022" has timed out for database "APS_SQL_DEV" after 10 seconds without a response. Check the service and network connections.
2007-09-06 15:07:43.20 Server Error: 1474, Severity: 16, State: 1.
2007-09-06 15:07:43.20 Server Database mirroring connection error 4 '64(The specified network name is no longer available.)' for 'TCP://db-server-2:5022'.
2007-09-06 15:08:06.03 spid9s Error: 1474, Severity: 16, State: 1.
2007-09-06 15:08:06.03 spid9s Database mirroring connection error 2 'Connection attempt failed with error: '10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)'.' for 'TCP://db-server-2:5022'.

hello,

from the logs it looks like you are not using the fully-qualified domain name (fqdn) when you are establishing the mirroring sessions. this is most likely the reason for the lack of connectivity.

when all the partners are up, you can use the sys.database_mirroring views on the principal and the mirror to figure out whether they both can talk to the witness (when all three nodes are up). if all connections are up, then automatic failover should happen. in your case most probably the mirror's connection to the witness will be shown as "disconnected", but if you use the fqdns it should work.

hth,

kaloian.

|||Trying giving the fully qualified domain name and see if it works as shown below,

IPCONFIG /ALL
Concatenate the "Host Name" and "Primary DNS Suffix". If you see something like:
Host Name . . . . . . . . . . . . : A
Primary Dns Suffix . . . . . . . : corp.mycompany.com

Then the computer name is just A.corp.mycompany.com. Prefix 'TCP://' and append ':' and you then have the partner name.
On the mirror server, you would just repeat the same command, but with the principal server named :

ALTER DATABASE [AdventureWorks] SET PARTNER =
N'TCP://A.corp.mycompany.com:5022'

On the principal server, you next specify the witness server:

ALTER DATABASE [AdventureWorks] SET WITNESS =
N'TCP://W.corp.mycompany.com:5026'

|||

Thank for replying. I appreciate it very much.

I had to add the FQDN in the host file in Windows->system32->drivers->etc->host on the mirror server and principal server inatnaces and I restarted the witness server after I setup failover. Once I did all that...it worked.

Thanks Again!

Chris

Automatic Failover Problem

Setup Configuration:

3 servers
- PRINCIPAL IP: 10.2.5.31 - DNS Lookup: db-server-2.mosside.choruscall.com
- MIRROR IP: 10.2.5.30 - DNS Lookup: sql-mirror.mosside.choruscall.com
- WITNESS ip: 10.2.5.32 - DNS Lookup: sql-witness.mosside.choruscall.com

Each Server is running Windows Server 2003 Enterprise Edition with SQL Server 2005 Enterprise Edition.
All server instances are enabled for remote connections(By default they are not).
All servers have the flag 1400 traceon and have been restarted.
PORT 5022 is unrestricted on network.

The server instances are connecting via certificates. Each server has an endpoint for the certificates to to connect on.

Certificate Setup Proceedure:

Principal_Host:
1. Create Master Key with Password

2. Create certificate with subject

3. Create endpoint for certificate (Listener_Port = 5022, Listener_ip = all)
to connect on for database_mirroring

4. Backup Certificate (principal_cert.cer)

5. Take backed up certificate to Mirror_Host

(Reapeat Steps 1-5 for Witness and Mirror)

Mirror_Host: Create Certificate on Mirror_Host for inbound connections from Principal:

6.(On Mirror_Host) Create Login for Principal using same password in step 1 (principal_login)

7. Create user for login just created. (principal_user)

8. Create local certificate for Principal on Mirror using certificate generated by principal.

ex: Create Certificate Principal_cert Authorization Principal_user FROM FILE='c:\principal_cert.cer'

9. (If an endpoint has been created already on the mirror)Grant connectiion to the login:

ex: Grant connect on endpoint::mirror_endpoint to principal_login

Repeat Steaps 6-9 for Principal and Witness Servers accordingly.

10. Import Database to SQL Server 2005 Principal Instance

11. Backup Database to disk with format

12. Backup Database log file to disk with format

13. Copy backups to mirror

14. Restore Database and log file with norecovery on Mirror_Host

15. Configre Database for Database Mirroring on Principal Server
There are two ways to do this. Via the wizzard or via the Transact-SQL window.
Using the wizzard appears to work since I started using FQDN.

PROBLEM:

After configuration, everythig appears to be correct. That is, the principal displays
that it is the principal and it is synchronized with the mirror. The mirror also displays that it is the
mirror and it is synchronized with the principal and it is in recovery. If I failover manually, the mirror
becomes the principal and the principal becomes the mirror (They form a quarum). If I disconnect the principal
from the network, the mirror is supposed to form a quarum with the witness and promote itself to principal status.
This is not what is happening. The witness recognizes that the principal is down and logs that info into its log file.
The Mirror attempts to contact the witness but cannot log onto the machine. The Mirror Logs the following:

Error: 1438, Severity: 16, State: 2.
The server instance Witness rejected configure request; read its error log file for more information.
The reason 1451, and state 3, can be of use for diagnostics by Microsoft.
This is a transient error hence retrying the request is likely to succeed.
Correct the cause if any and retry.

<<<<<<<MIRROR SERVER >>>>>>>>

2007-09-06 15:08:45.32 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:08:45.32 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:05.32 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:05.32 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:25.33 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:25.33 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:09:45.34 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:09:45.34 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:10:05.35 spid23s Error: 1438, Severity: 16, State: 2.
2007-09-06 15:10:05.35 spid23s The server instance Witness rejected configure request; read its error log file for more information. The reason 1451, and state 3, can be of use for diagnostics by Microsoft. This is a transient error hence retrying the request is likely to succeed. Correct the cause if any and retry.
2007-09-06 15:10:25.36 spid23s Error: 1438, Severity: 16, State: 2.

<<<<<<< WITNESS SERVER >>>>>>>>

2007-09-06 14:19:55.90 spid52 The Database Mirroring protocol transport is now listening for connections.
2007-09-06 15:07:11.64 spid24s Error: 1479, Severity: 16, State: 1.
2007-09-06 15:07:11.64 spid24s The mirroring connection to "TCP://db-server-2:5022" has timed out for database "APS_SQL_DEV" after 10 seconds without a response. Check the service and network connections.
2007-09-06 15:07:43.20 Server Error: 1474, Severity: 16, State: 1.
2007-09-06 15:07:43.20 Server Database mirroring connection error 4 '64(The specified network name is no longer available.)' for 'TCP://db-server-2:5022'.
2007-09-06 15:08:06.03 spid9s Error: 1474, Severity: 16, State: 1.
2007-09-06 15:08:06.03 spid9s Database mirroring connection error 2 'Connection attempt failed with error: '10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)'.' for 'TCP://db-server-2:5022'.

hello,

from the logs it looks like you are not using the fully-qualified domain name (fqdn) when you are establishing the mirroring sessions. this is most likely the reason for the lack of connectivity.

when all the partners are up, you can use the sys.database_mirroring views on the principal and the mirror to figure out whether they both can talk to the witness (when all three nodes are up). if all connections are up, then automatic failover should happen. in your case most probably the mirror's connection to the witness will be shown as "disconnected", but if you use the fqdns it should work.

hth,

kaloian.

|||Trying giving the fully qualified domain name and see if it works as shown below,

IPCONFIG /ALL
Concatenate the "Host Name" and "Primary DNS Suffix". If you see something like:
Host Name . . . . . . . . . . . . : A
Primary Dns Suffix . . . . . . . : corp.mycompany.com

Then the computer name is just A.corp.mycompany.com. Prefix 'TCP://' and append ':' and you then have the partner name.
On the mirror server, you would just repeat the same command, but with the principal server named :

ALTER DATABASE [AdventureWorks] SET PARTNER =
N'TCP://A.corp.mycompany.com:5022'

On the principal server, you next specify the witness server:

ALTER DATABASE [AdventureWorks] SET WITNESS =
N'TCP://W.corp.mycompany.com:5026'

|||

Thank for replying. I appreciate it very much.

I had to add the FQDN in the host file in Windows->system32->drivers->etc->host on the mirror server and principal server inatnaces and I restarted the witness server after I setup failover. Once I did all that...it worked.

Thanks Again!

Chris

Automatic failover failure

Hello,

we're running a mirrored database with High Availability for Automatic failover including a Witness instance for a web application.

When doing a manual failover on the database in Management studio, the roles are switched correctly and the database is in "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. The web application has no problems switching servers by using client failover with the jdbc driver. There is no problem accessing the database with Management Studio.

However, if we stop the SQL service on the Principal server the role is automatically failed over to the Mirror server by the Witness. The database is then in the mode "Principal, Disconnected" which should be fine. However, accessing the database from the web application or with Management Studio yields some strange results. It is not possible to write to the database, and reading from the database works inconsistently (the web application seems like it can do it, but not from the Management Studio).

Starting the SQL service on the former Principal server makes the database go into mode "Mirror, Synchronizing/Restoring" and "Principal, Synchronizing". And it will stay that way indefinitely. There are not that many updates/transactions made to the database that can make it stay in this state, especially if you can't write to the database in the first place.

The next step taken after being stuck in this state is to stop the SQL service on the Mirror (former Principal), restart the service on the Principal (former Mirror). Accessing the database now works. The database is in mode "Principal, Disconnected". Starting the SQL service on the Mirror (former Principal) makes the database go into the normal "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. Access to database is normal.

The same erroneous behaviour can be observed by unplugging the network cable on the Principal server, so it seems like we can only get a smooth transition by doing a manual failover.

Any ideas on what might be the problem? Has anybody experienced a similar situation?

Can you see any errors or warnings from SQL error log in Principal & mirror servers?|||

These are entires I've found on the principal (former mirror) server

2006-11-26 12:42:52.60 spid8s Database mirroring is inactive for database 'db'. This is an informational message only. No user action is required.
2006-11-26 12:42:52.60 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.60 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:52.60 spid8s The mirrored database "db" is changing roles from "MIRROR" to "PRINCIPAL" due to Auto Failover.
2006-11-26 12:42:52.75 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.75 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:54.52 spid8s Recovery is writing a checkpoint in database 'db' (5). This is an informational message only. No user action is required.

These are entries I've found on the witness server

2006-11-26 12:45:03.36 spid27s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.36 spid27s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.50 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.50 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.82 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.82 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:04.03 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:04.03 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.

I guess the errors about receiving data from SERVER01 should be normal since the SQL service is stopped on that server during this time.

Thanks,

Ingmar

|||

We are experiencing the same issue.

Some additional information on our situation:

We are running a mirrored database with High Availability for Automatic failover including a Witness instance for a non web application. As in the original post we are stuck (Principal, Synchronizing) and (Mirror Synchronizing / Restoring...).

We are using the new SqlCacheDependency (command based vs. table based polling) cache notification.

We are able to access some of the data in the database via SQL Server Management Studio - query window, however we can not expand the database in the Object Explorer window and if we right click on the database and attempt to access the properties page we receive a 'Lock request time out period exceeded. (Microsoft SQL Server, Error: 1222)'.

This may not be pertinent but we also noticed that even though the query from sys.database_mirroring show that the databases are in sync we observe that the Modified and Accessed dates on the new Principal .mdf and .ldf files lags the new Mirror partner by 16 Hrs for the .ldf file and 7 Days for the .mdf file.

|||Check this KBA http://support.microsoft.com/kb/912422 is any help in this regard.|||

Hello,

We were able to reproduce the problem in a test environment and it seems that we're only having problems with databases that have fulltext indexes. And, after applying the SP2 CTP in this environment the problem went away. We haven't touched the production servers yet as we wait for the 'full' release of the SP2.

Regards,

Ingmar

|||

We are also experiencing the exact same problems with trying to mirror a database containing a Full Text index. Does anybody have an estimate as to when the "full" release of SP2 is to be released?

Thanks,

Jody

Automatic failover failure

Hello,

we're running a mirrored database with High Availability for Automatic failover including a Witness instance for a web application.

When doing a manual failover on the database in Management studio, the roles are switched correctly and the database is in "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. The web application has no problems switching servers by using client failover with the jdbc driver. There is no problem accessing the database with Management Studio.

However, if we stop the SQL service on the Principal server the role is automatically failed over to the Mirror server by the Witness. The database is then in the mode "Principal, Disconnected" which should be fine. However, accessing the database from the web application or with Management Studio yields some strange results. It is not possible to write to the database, and reading from the database works inconsistently (the web application seems like it can do it, but not from the Management Studio).

Starting the SQL service on the former Principal server makes the database go into mode "Mirror, Synchronizing/Restoring" and "Principal, Synchronizing". And it will stay that way indefinitely. There are not that many updates/transactions made to the database that can make it stay in this state, especially if you can't write to the database in the first place.

The next step taken after being stuck in this state is to stop the SQL service on the Mirror (former Principal), restart the service on the Principal (former Mirror). Accessing the database now works. The database is in mode "Principal, Disconnected". Starting the SQL service on the Mirror (former Principal) makes the database go into the normal "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. Access to database is normal.

The same erroneous behaviour can be observed by unplugging the network cable on the Principal server, so it seems like we can only get a smooth transition by doing a manual failover.

Any ideas on what might be the problem? Has anybody experienced a similar situation?

Can you see any errors or warnings from SQL error log in Principal & mirror servers?|||

These are entires I've found on the principal (former mirror) server

2006-11-26 12:42:52.60 spid8s Database mirroring is inactive for database 'db'. This is an informational message only. No user action is required.
2006-11-26 12:42:52.60 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.60 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:52.60 spid8s The mirrored database "db" is changing roles from "MIRROR" to "PRINCIPAL" due to Auto Failover.
2006-11-26 12:42:52.75 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.75 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:54.52 spid8s Recovery is writing a checkpoint in database 'db' (5). This is an informational message only. No user action is required.

These are entries I've found on the witness server

2006-11-26 12:45:03.36 spid27s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.36 spid27s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.50 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.50 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.82 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.82 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:04.03 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:04.03 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.

I guess the errors about receiving data from SERVER01 should be normal since the SQL service is stopped on that server during this time.

Thanks,

Ingmar

|||

We are experiencing the same issue.

Some additional information on our situation:

We are running a mirrored database with High Availability for Automatic failover including a Witness instance for a non web application. As in the original post we are stuck (Principal, Synchronizing) and (Mirror Synchronizing / Restoring...).

We are using the new SqlCacheDependency (command based vs. table based polling) cache notification.

We are able to access some of the data in the database via SQL Server Management Studio - query window, however we can not expand the database in the Object Explorer window and if we right click on the database and attempt to access the properties page we receive a 'Lock request time out period exceeded. (Microsoft SQL Server, Error: 1222)'.

This may not be pertinent but we also noticed that even though the query from sys.database_mirroring show that the databases are in sync we observe that the Modified and Accessed dates on the new Principal .mdf and .ldf files lags the new Mirror partner by 16 Hrs for the .ldf file and 7 Days for the .mdf file.

|||Check this KBA http://support.microsoft.com/kb/912422 is any help in this regard.|||

Hello,

We were able to reproduce the problem in a test environment and it seems that we're only having problems with databases that have fulltext indexes. And, after applying the SP2 CTP in this environment the problem went away. We haven't touched the production servers yet as we wait for the 'full' release of the SP2.

Regards,

Ingmar

|||

We are also experiencing the exact same problems with trying to mirror a database containing a Full Text index. Does anybody have an estimate as to when the "full" release of SP2 is to be released?

Thanks,

Jody

Automatic failover failure

Hello,

we're running a mirrored database with High Availability for Automatic failover including a Witness instance for a web application.

When doing a manual failover on the database in Management studio, the roles are switched correctly and the database is in "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. The web application has no problems switching servers by using client failover with the jdbc driver. There is no problem accessing the database with Management Studio.

However, if we stop the SQL service on the Principal server the role is automatically failed over to the Mirror server by the Witness. The database is then in the mode "Principal, Disconnected" which should be fine. However, accessing the database from the web application or with Management Studio yields some strange results. It is not possible to write to the database, and reading from the database works inconsistently (the web application seems like it can do it, but not from the Management Studio).

Starting the SQL service on the former Principal server makes the database go into mode "Mirror, Synchronizing/Restoring" and "Principal, Synchronizing". And it will stay that way indefinitely. There are not that many updates/transactions made to the database that can make it stay in this state, especially if you can't write to the database in the first place.

The next step taken after being stuck in this state is to stop the SQL service on the Mirror (former Principal), restart the service on the Principal (former Mirror). Accessing the database now works. The database is in mode "Principal, Disconnected". Starting the SQL service on the Mirror (former Principal) makes the database go into the normal "Principal, Synchronized" and "Mirror, Synchronized/Restoring" mode. Access to database is normal.

The same erroneous behaviour can be observed by unplugging the network cable on the Principal server, so it seems like we can only get a smooth transition by doing a manual failover.

Any ideas on what might be the problem? Has anybody experienced a similar situation?

Can you see any errors or warnings from SQL error log in Principal & mirror servers?|||

These are entires I've found on the principal (former mirror) server

2006-11-26 12:42:52.60 spid8s Database mirroring is inactive for database 'db'. This is an informational message only. No user action is required.
2006-11-26 12:42:52.60 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.60 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:52.60 spid8s The mirrored database "db" is changing roles from "MIRROR" to "PRINCIPAL" due to Auto Failover.
2006-11-26 12:42:52.75 spid22s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:42:52.75 spid22s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:42:54.52 spid8s Recovery is writing a checkpoint in database 'db' (5). This is an informational message only. No user action is required.

These are entries I've found on the witness server

2006-11-26 12:45:03.36 spid27s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.36 spid27s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.50 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.50 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:03.82 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:03.82 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.
2006-11-26 12:45:04.03 spid19s Error: 1474, Severity: 16, State: 1.
2006-11-26 12:45:04.03 spid19s Database mirroring connection error 4 'An error occurred while receiving data: '64(error not found)'.' for 'TCP://SERVER01:7024'.

I guess the errors about receiving data from SERVER01 should be normal since the SQL service is stopped on that server during this time.

Thanks,

Ingmar

|||

We are experiencing the same issue.

Some additional information on our situation:

We are running a mirrored database with High Availability for Automatic failover including a Witness instance for a non web application. As in the original post we are stuck (Principal, Synchronizing) and (Mirror Synchronizing / Restoring...).

We are using the new SqlCacheDependency (command based vs. table based polling) cache notification.

We are able to access some of the data in the database via SQL Server Management Studio - query window, however we can not expand the database in the Object Explorer window and if we right click on the database and attempt to access the properties page we receive a 'Lock request time out period exceeded. (Microsoft SQL Server, Error: 1222)'.

This may not be pertinent but we also noticed that even though the query from sys.database_mirroring show that the databases are in sync we observe that the Modified and Accessed dates on the new Principal .mdf and .ldf files lags the new Mirror partner by 16 Hrs for the .ldf file and 7 Days for the .mdf file.

|||Check this KBA http://support.microsoft.com/kb/912422 is any help in this regard.|||

Hello,

We were able to reproduce the problem in a test environment and it seems that we're only having problems with databases that have fulltext indexes. And, after applying the SP2 CTP in this environment the problem went away. We haven't touched the production servers yet as we wait for the 'full' release of the SP2.

Regards,

Ingmar

|||

We are also experiencing the exact same problems with trying to mirror a database containing a Full Text index. Does anybody have an estimate as to when the "full" release of SP2 is to be released?

Thanks,

Jody

Automatic client redirection during the hot failover in .NET 1.1 applications and SQL 2005

Hi,
We would like to use SQL Server 2005 with our .NET 1.1 web applications.
We would like to leverage the new mirroring enhancements to achieve higher
server availability.
However, I know that .NET 1.1 doesn't support automatic client redirection
during the hot failover
and the only option is to handle the redirection manually in the code. By
saying that I mean the following:
string connectionString="Trusted_Connection=Yes;Data Source=SERVER1;Initial
Catalog=MirrorTest;Integrated Security=SSPI;";
SqlConnection connection=new SqlConnection(connectionString);
try
{
connection.Open();
}
catch
{
connection.ConnectionString=connectionString.repla ce("Data
Source=SERVER1","Data Source=SERVER2");
connection.Open();
}
I saw that someone has suggested to use the .NET OdbcClient or OleDB client
and have them access the new SQL Native Client library which is
mirroring-aware
([url]http://groups.google.com/group/microsoft.public.sqlserver.programming/browse_thread/thread/90517df320d22f1/cc09960826a31a94?lnk=st&q=sql+server+2005+Failover +.net+1.1&rnum=10&hl=en#cc09960826a31a94)[/url].
So my question is what is the better way to use and how the second way can
be implemented? Does it mean that we need to download SQL Native Client from
SQL
2005 feature pack?
Thank you very much,
Best Regards
Michael
Hi Michael,
My understanding of your issue is that:
You would like to know how to use the OLEDB provider, ODBC.NET or ADO with
SQL Native Client so that you can directly configure the connection string
for the database mirroring.
If I have misunderstood, please let me know.
Yes, you need to install SQL Native Client first on your computer before
further actions. SQL Native Client combines the SQL OLD DB provider and the
SQL ODBC driver into one native DLL while also providing new functionality
above and beyond that supplied by the MDAC, so it is easy to be integrated
to old applications.
You may refer to:
SQL Native Client Programming
http://msdn2.microsoft.com/en-us/library/ms130892.aspx
The connection string for database mirroring is "server=Partner_A; failover
partner=Partner_B; database=AdventureWorks". You may refer to:
Making the Initial Connection to a Database Mirroring Session
http://msdn2.microsoft.com/en-us/library/ms366348.aspx
Using Connection String Keywords with SQL Native Client
http://msdn2.microsoft.com/en-us/library/ms130822.aspx
Hope this helps. Please feel free to let me know if you need further
assistance.
Charles Wang
Microsoft Online Community Support
================================================== ====
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
================================================== ====
This posting is provided "AS IS" with no warranties, and confers no rights.
================================================== ====

Automatic client redirection during the hot failover in .NET 1.1 applications and SQL 2005

Hi,
We would like to use SQL Server 2005 with our .NET 1.1 web applications.
We would like to leverage the new mirroring enhancements to achieve higher
server availability.
However, I know that .NET 1.1 doesn't support automatic client redirection
during the hot failover
and the only option is to handle the redirection manually in the code. By
saying that I mean the following:
string connectionString="Trusted_Connection=Yes;Data Source=SERVER1;Initial
Catalog=MirrorTest;Integrated Security=SSPI;";
SqlConnection connection=new SqlConnection(connectionString);
try
{
connection.Open();
}
catch
{
connection.ConnectionString=connectionString.replace("Data
Source=SERVER1","Data Source=SERVER2");
connection.Open();
}
I saw that someone has suggested to use the .NET OdbcClient or OleDB client
and have them access the new SQL Native Client library which is
mirroring-aware
(http://groups.google.com/group/microsoft.public.sqlserver.programming/browse_thread/thread/90517df320d22f1/cc09960826a31a94?lnk=st&q=sql+server+2005+Failover+.net+1.1&rnum=10&hl=en#cc09960826a31a94).
So my question is what is the better way to use and how the second way can
be implemented? Does it mean that we need to download SQL Native Client from
SQL
2005 feature pack?
Thank you very much,
Best Regards
MichaelHi Michael,
My understanding of your issue is that:
You would like to know how to use the OLEDB provider, ODBC.NET or ADO with
SQL Native Client so that you can directly configure the connection string
for the database mirroring.
If I have misunderstood, please let me know.
Yes, you need to install SQL Native Client first on your computer before
further actions. SQL Native Client combines the SQL OLD DB provider and the
SQL ODBC driver into one native DLL while also providing new functionality
above and beyond that supplied by the MDAC, so it is easy to be integrated
to old applications.
You may refer to:
SQL Native Client Programming
http://msdn2.microsoft.com/en-us/library/ms130892.aspx
The connection string for database mirroring is "server=Partner_A; failover
partner=Partner_B; database=AdventureWorks". You may refer to:
Making the Initial Connection to a Database Mirroring Session
http://msdn2.microsoft.com/en-us/library/ms366348.aspx
Using Connection String Keywords with SQL Native Client
http://msdn2.microsoft.com/en-us/library/ms130822.aspx
Hope this helps. Please feel free to let me know if you need further
assistance.
Charles Wang
Microsoft Online Community Support
======================================================When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================