Thursday, March 29, 2012
Automating Restoring of *.BAK files
create flat file backup of full databases to *.BAK files nightly.
Is it possible to automate the restoring of such BAK files on another SQL
Server 2000 SP3a on another server (assume I have in place scripts for
copying the BAK files from the source server to the destination server)? If
so, how?http://msdn.microsoft.com/library/en-us/adminsql/ad_automate_42r7.asp
--
David Portas
SQL Server MVP
--|||Yes, I know how to create a job in general, but what exactly do I run to
restore a BAK file?
"David Portas" <REMOVE_BEFORE_REPLYING_dportas@.acm.org> wrote in message
news:1106572451.019746.272450@.f14g2000cwb.googlegroups.com...
> http://msdn.microsoft.com/library/en-us/adminsql/ad_automate_42r7.asp
> --
> David Portas
> SQL Server MVP
> --
>|||Use the RESTORE DATABASE command in a Transact SQL job step. See Books
Online for details of the RESTORE DATABASE command.
--
David Portas
SQL Server MVP
--|||Taking a step back, I am just wondering whether a flat-file backup-restore
would be the best way to synchronise 2 SQL Server 2000 databases? Or should
I go for a DTS package to export database on the source server to an Access
mdb file and import it on the other end? Sometimes, I find that the users
in an exported flat file, following an import on another server is not
"usable" even if the referenced user are already defined on the destination
server.
"Patrick" <patl@.reply.newsgroup.msn.com> wrote in message
news:%23CWx88hAFHA.2552@.TK2MSFTNGP09.phx.gbl...
> Yes, I know how to create a job in general, but what exactly do I run to
> restore a BAK file?
>
> "David Portas" <REMOVE_BEFORE_REPLYING_dportas@.acm.org> wrote in message
> news:1106572451.019746.272450@.f14g2000cwb.googlegroups.com...
> > http://msdn.microsoft.com/library/en-us/adminsql/ad_automate_42r7.asp
> > --
> > David Portas
> > SQL Server MVP
> > --
> >
>|||"Patrick" <patl@.reply.newsgroup.msn.com> wrote in message
news:%23S0tVOhAFHA.3416@.TK2MSFTNGP09.phx.gbl...
> I have set up a Maintenance Plans on a SQL Server 2000 SP3a on one server
to
> create flat file backup of full databases to *.BAK files nightly.
> Is it possible to automate the restoring of such BAK files on another SQL
> Server 2000 SP3a on another server (assume I have in place scripts for
> copying the BAK files from the source server to the destination server)?
If
> so, how?
>
Yes.
In my case I wrote a stored proc on the restoring server and called it from
the backing up server.
CREATE procedure restore_FOO as
declare @.backup_file as varchar(255)
select @.backup_file=physical_device_name from
nell.msdb.dbo.backupmediafamily where media_set_id in (select
max(media_set_id) from BAR.msdb.dbo.backupset where database_name='foo')
print @.backup_file
restore database FOO from disk=@.backup_file with
move 'SearchActivity_Data' to 'e:\sql_data\FOO_data.mdf',
move 'SearchActivity_Log' to 'f:\SQL_LOGs\FOO_log.ldf',
move 'SearchActivity_Index' to 'g:\sql_index\FOO_Index_Data.NDF',
replace
GO
>
Automating Restoring of *.BAK files
create flat file backup of full databases to *.BAK files nightly.
Is it possible to automate the restoring of such BAK files on another SQL
Server 2000 SP3a on another server (assume I have in place scripts for
copying the BAK files from the source server to the destination server)? If
so, how?
http://msdn.microsoft.com/library/en...omate_42r7.asp
David Portas
SQL Server MVP
|||Yes, I know how to create a job in general, but what exactly do I run to
restore a BAK file?
"David Portas" <REMOVE_BEFORE_REPLYING_dportas@.acm.org> wrote in message
news:1106572451.019746.272450@.f14g2000cwb.googlegr oups.com...
> http://msdn.microsoft.com/library/en...omate_42r7.asp
> --
> David Portas
> SQL Server MVP
> --
>
|||Use the RESTORE DATABASE command in a Transact SQL job step. See Books
Online for details of the RESTORE DATABASE command.
David Portas
SQL Server MVP
|||"Patrick" <patl@.reply.newsgroup.msn.com> wrote in message
news:%23S0tVOhAFHA.3416@.TK2MSFTNGP09.phx.gbl...
> I have set up a Maintenance Plans on a SQL Server 2000 SP3a on one server
to
> create flat file backup of full databases to *.BAK files nightly.
> Is it possible to automate the restoring of such BAK files on another SQL
> Server 2000 SP3a on another server (assume I have in place scripts for
> copying the BAK files from the source server to the destination server)?
If
> so, how?
>
Yes.
In my case I wrote a stored proc on the restoring server and called it from
the backing up server.
CREATE procedure restore_FOO as
declare @.backup_file as varchar(255)
select @.backup_file=physical_device_name from
nell.msdb.dbo.backupmediafamily where media_set_id in (select
max(media_set_id) from BAR.msdb.dbo.backupset where database_name='foo')
print @.backup_file
restore database FOO from disk=@.backup_file with
move 'SearchActivity_Data' to 'e:\sql_data\FOO_data.mdf',
move 'SearchActivity_Log' to 'f:\SQL_LOGs\FOO_log.ldf',
move 'SearchActivity_Index' to 'g:\sql_index\FOO_Index_Data.NDF',
replace
GO
>
sql
Automating Restoring of *.BAK files
create flat file backup of full databases to *.BAK files nightly.
Is it possible to automate the restoring of such BAK files on another SQL
Server 2000 SP3a on another server (assume I have in place scripts for
copying the BAK files from the source server to the destination server)? If
so, how?http://msdn.microsoft.com/library/e...tomate_42r7.asp
--
David Portas
SQL Server MVP
--|||Yes, I know how to create a job in general, but what exactly do I run to
restore a BAK file?
"David Portas" <REMOVE_BEFORE_REPLYING_dportas@.acm.org> wrote in message
news:1106572451.019746.272450@.f14g2000cwb.googlegroups.com...
> http://msdn.microsoft.com/library/e...tomate_42r7.asp
> --
> David Portas
> SQL Server MVP
> --
>|||Use the RESTORE DATABASE command in a Transact SQL job step. See Books
Online for details of the RESTORE DATABASE command.
David Portas
SQL Server MVP
--|||"Patrick" <patl@.reply.newsgroup.msn.com> wrote in message
news:%23S0tVOhAFHA.3416@.TK2MSFTNGP09.phx.gbl...
> I have set up a Maintenance Plans on a SQL Server 2000 SP3a on one server
to
> create flat file backup of full databases to *.BAK files nightly.
> Is it possible to automate the restoring of such BAK files on another SQL
> Server 2000 SP3a on another server (assume I have in place scripts for
> copying the BAK files from the source server to the destination server)?
If
> so, how?
>
Yes.
In my case I wrote a stored proc on the restoring server and called it from
the backing up server.
CREATE procedure restore_FOO as
declare @.backup_file as varchar(255)
select @.backup_file=physical_device_name from
nell.msdb.dbo.backupmediafamily where media_set_id in (select
max(media_set_id) from BAR.msdb.dbo.backupset where database_name='foo')
print @.backup_file
restore database FOO from disk=@.backup_file with
move 'SearchActivity_Data' to 'e:\sql_data\FOO_data.mdf',
move 'SearchActivity_Log' to 'f:\SQL_LOGs\FOO_log.ldf',
move 'SearchActivity_Index' to 'g:\sql_index\FOO_Index_Data.NDF',
replace
GO
>
Automating report execution - saving to separate files
I'm trying to do something which I hope can be accomplished relatively simply.
I have a report similar to bank statements let's say. When run, it currently prints out each person's statement into one file, with page breaks sepearating each person's statement. What I need to do, is when the report is run, save each person's report into a seperate file for the purpose of emailing to them later.
I could easily modify my report to just output for one particular person, but I'm not sure if there's a way to "bulk render" all the reports and have them saved to sepearate files.
I should also add that I'm using an MS Access Data Project (ADP) as the front end to my app - connected to a SQL Server 2005 DB. I currently display the reports by embedding a web browser object into an Access form and rendering the report via HTML.
Thanks in advance,
H
I actually figured out an answer to my own question which I'll post here in case others need to do the same thing. If anyone else has an alternative suggestion please feel free to let me know.
There's a VBA function called "URLDownloadToFile". If I use the reporting services URL access and specify type XLS or PDF, I can pass this as the URL and save the file to a specific folder. Kind of nice actually - and very simple.
Tuesday, March 27, 2012
automatically stop/restart sql service
2005 express, use a script to transfer files over to another server and
restart the service automatically.
Any ideas of how I can do this?
You can use a batch file that does a net stop, copy and net start. Schedule
it with task scheduler.
Jason Massie
Web: http://statisticsio.com
RSS: http://feeds.feedburner.com/statisticsio
"pete0085" <pete0085@.discussions.microsoft.com> wrote in message
news:63C59DBE-8BBF-4967-BFB4-EDEF3DA0B84C@.microsoft.com...
>I would like to be able to automatically stop a sql service running on sql
> 2005 express, use a script to transfer files over to another server and
> restart the service automatically.
> Any ideas of how I can do this?
|||When it does a copy and it's only copying one file, not the entire directory.
The directory has 4 folders and 1 readme file and it's copying the readme
file.
Any ideas?
"Jason Massie" wrote:
> You can use a batch file that does a net stop, copy and net start. Schedule
> it with task scheduler.
>
> --
> Jason Massie
> Web: http://statisticsio.com
> RSS: http://feeds.feedburner.com/statisticsio
> "pete0085" <pete0085@.discussions.microsoft.com> wrote in message
> news:63C59DBE-8BBF-4967-BFB4-EDEF3DA0B84C@.microsoft.com...
>
>
|||xcopy /S
Documentation for the other switches is available by running xcopy /?
Jason Massie
Web: http://statisticsio.com
RSS: http://feeds.feedburner.com/statisticsio
"pete0085" <pete0085@.discussions.microsoft.com> wrote in message
news:752C5BDF-A76C-48E3-8D9F-013739AE87AE@.microsoft.com...[vbcol=seagreen]
> When it does a copy and it's only copying one file, not the entire
> directory.
> The directory has 4 folders and 1 readme file and it's copying the readme
> file.
> Any ideas?
> "Jason Massie" wrote:
sql
automatically stop/restart sql service
2005 express, use a script to transfer files over to another server and
restart the service automatically.
Any ideas of how I can do this?You can use a batch file that does a net stop, copy and net start. Schedule
it with task scheduler.
Jason Massie
Web: http://statisticsio.com
RSS: http://feeds.feedburner.com/statisticsio
"pete0085" <pete0085@.discussions.microsoft.com> wrote in message
news:63C59DBE-8BBF-4967-BFB4-EDEF3DA0B84C@.microsoft.com...
>I would like to be able to automatically stop a sql service running on sql
> 2005 express, use a script to transfer files over to another server and
> restart the service automatically.
> Any ideas of how I can do this?|||When it does a copy and it's only copying one file, not the entire directory.
The directory has 4 folders and 1 readme file and it's copying the readme
file.
Any ideas?
"Jason Massie" wrote:
> You can use a batch file that does a net stop, copy and net start. Schedule
> it with task scheduler.
>
> --
> Jason Massie
> Web: http://statisticsio.com
> RSS: http://feeds.feedburner.com/statisticsio
> "pete0085" <pete0085@.discussions.microsoft.com> wrote in message
> news:63C59DBE-8BBF-4967-BFB4-EDEF3DA0B84C@.microsoft.com...
> >I would like to be able to automatically stop a sql service running on sql
> > 2005 express, use a script to transfer files over to another server and
> > restart the service automatically.
> >
> > Any ideas of how I can do this?
>
>|||xcopy /S
Documentation for the other switches is available by running xcopy /?
--
Jason Massie
Web: http://statisticsio.com
RSS: http://feeds.feedburner.com/statisticsio
"pete0085" <pete0085@.discussions.microsoft.com> wrote in message
news:752C5BDF-A76C-48E3-8D9F-013739AE87AE@.microsoft.com...
> When it does a copy and it's only copying one file, not the entire
> directory.
> The directory has 4 folders and 1 readme file and it's copying the readme
> file.
> Any ideas?
> "Jason Massie" wrote:
>> You can use a batch file that does a net stop, copy and net start.
>> Schedule
>> it with task scheduler.
>>
>> --
>> Jason Massie
>> Web: http://statisticsio.com
>> RSS: http://feeds.feedburner.com/statisticsio
>> "pete0085" <pete0085@.discussions.microsoft.com> wrote in message
>> news:63C59DBE-8BBF-4967-BFB4-EDEF3DA0B84C@.microsoft.com...
>> >I would like to be able to automatically stop a sql service running on
>> >sql
>> > 2005 express, use a script to transfer files over to another server and
>> > restart the service automatically.
>> >
>> > Any ideas of how I can do this?
>>
Thursday, March 22, 2012
Automatic XSD generation
Is there standard functionality to generate XSD files, which i can use
from within a program?
If so, please let me know where i can find more information.
Thanks
'I'm a raven, aren't I?' it said. 'One of the few birds who speak. The first
thing people say is, oh, you're a raven, go on, say the N word... If I had a
penny for every time that's happened, I'd-'
(Soul Music)
There are quite a few ways to handle XSDs.
From SQL, using XMLDATA attaches the Schema to your results. If you run a
query like:
SELECT * FROM Customers
FOR XML AUTO, XMLDATA
you will have a schema attached that is attribute centric. You can change to
elements like:
SELECT * FROM Customers
FOR XML AUTO, ELEMENTS, XMLDATA
But, you still have a schema attached to the top of an XML snippet (no root
tag). If you just want schema, you end up having to remove the records.
Something like:
SELECT * FROM Customers
WHERE 1 = 0
FOR XML AUTO, ELEMENTS, XMLDATA
You are still missing root tag, however, which makes the XSD only
semi-useful, IMO.
With .NET, you can create a DataSet and return the XML Schema. This creates
an XSD that is valid. BUT, you end up with the extra weight of the DataSet
tags that surround your table of records. This is not a huge burden, overall,
but it can add a bit of extra work to those consuming your XML outside of the
..NET platform. It is, overall, easier than working directly with the SQL
"autogen" capabilities.
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
***************************
Think Outside the Box!
***************************
"Mark Pustjens" wrote:
> Hello everyone,
> Is there standard functionality to generate XSD files, which i can use
> from within a program?
> If so, please let me know where i can find more information.
> Thanks
> --
> 'I'm a raven, aren't I?' it said. 'One of the few birds who speak. The first
> thing people say is, oh, you're a raven, go on, say the N word... If I had a
> penny for every time that's happened, I'd-'
> (Soul Music)
>
|||Note that the XMLDATA directive generates the older XDR format and not the
XSD format.
Best regards
Michael
"Cowboy (Gregory A. Beamer) - MVP" <NoSpamMgbworld@.comcast.netNoSpamM> wrote
in message news:0888E260-B1EC-49D5-B7BE-0FD4A383156E@.microsoft.com...[vbcol=seagreen]
> There are quite a few ways to handle XSDs.
> From SQL, using XMLDATA attaches the Schema to your results. If you run a
> query like:
> SELECT * FROM Customers
> FOR XML AUTO, XMLDATA
> you will have a schema attached that is attribute centric. You can change
> to
> elements like:
> SELECT * FROM Customers
> FOR XML AUTO, ELEMENTS, XMLDATA
> But, you still have a schema attached to the top of an XML snippet (no
> root
> tag). If you just want schema, you end up having to remove the records.
> Something like:
> SELECT * FROM Customers
> WHERE 1 = 0
> FOR XML AUTO, ELEMENTS, XMLDATA
> You are still missing root tag, however, which makes the XSD only
> semi-useful, IMO.
> With .NET, you can create a DataSet and return the XML Schema. This
> creates
> an XSD that is valid. BUT, you end up with the extra weight of the DataSet
> tags that surround your table of records. This is not a huge burden,
> overall,
> but it can add a bit of extra work to those consuming your XML outside of
> the
> .NET platform. It is, overall, easier than working directly with the SQL
> "autogen" capabilities.
>
> --
> Gregory A. Beamer
> MVP; MCP: +I, SE, SD, DBA
> ***************************
> Think Outside the Box!
> ***************************
> "Mark Pustjens" wrote:
Automatic XSD generation
Is there standard functionality to generate XSD files, which i can use
from within a program?
If so, please let me know where i can find more information.
Thanks
'I'm a raven, aren't I?' it said. 'One of the few birds who speak. The first
thing people say is, oh, you're a raven, go on, say the N word... If I had a
penny for every time that's happened, I'd-'
(Soul Music)There are quite a few ways to handle XSDs.
From SQL, using XMLDATA attaches the Schema to your results. If you run a
query like:
SELECT * FROM Customers
FOR XML AUTO, XMLDATA
you will have a schema attached that is attribute centric. You can change to
elements like:
SELECT * FROM Customers
FOR XML AUTO, ELEMENTS, XMLDATA
But, you still have a schema attached to the top of an XML snippet (no root
tag). If you just want schema, you end up having to remove the records.
Something like:
SELECT * FROM Customers
WHERE 1 = 0
FOR XML AUTO, ELEMENTS, XMLDATA
You are still missing root tag, however, which makes the XSD only
semi-useful, IMO.
With .NET, you can create a DataSet and return the XML Schema. This creates
an XSD that is valid. BUT, you end up with the extra weight of the DataSet
tags that surround your table of records. This is not a huge burden, overall
,
but it can add a bit of extra work to those consuming your XML outside of th
e
.NET platform. It is, overall, easier than working directly with the SQL
"autogen" capabilities.
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
***************************
Think Outside the Box!
***************************
"Mark Pustjens" wrote:
> Hello everyone,
> Is there standard functionality to generate XSD files, which i can use
> from within a program?
> If so, please let me know where i can find more information.
> Thanks
> --
> 'I'm a raven, aren't I?' it said. 'One of the few birds who speak. The fir
st
> thing people say is, oh, you're a raven, go on, say the N word... If I had
a
> penny for every time that's happened, I'd-'
> (Soul Music)
>|||Note that the XMLDATA directive generates the older XDR format and not the
XSD format.
Best regards
Michael
"Cowboy (Gregory A. Beamer) - MVP" <NoSpamMgbworld@.comcast.netNoSpamM> wrote
in message news:0888E260-B1EC-49D5-B7BE-0FD4A383156E@.microsoft.com...
> There are quite a few ways to handle XSDs.
> From SQL, using XMLDATA attaches the Schema to your results. If you run a
> query like:
> SELECT * FROM Customers
> FOR XML AUTO, XMLDATA
> you will have a schema attached that is attribute centric. You can change
> to
> elements like:
> SELECT * FROM Customers
> FOR XML AUTO, ELEMENTS, XMLDATA
> But, you still have a schema attached to the top of an XML snippet (no
> root
> tag). If you just want schema, you end up having to remove the records.
> Something like:
> SELECT * FROM Customers
> WHERE 1 = 0
> FOR XML AUTO, ELEMENTS, XMLDATA
> You are still missing root tag, however, which makes the XSD only
> semi-useful, IMO.
> With .NET, you can create a DataSet and return the XML Schema. This
> creates
> an XSD that is valid. BUT, you end up with the extra weight of the DataSet
> tags that surround your table of records. This is not a huge burden,
> overall,
> but it can add a bit of extra work to those consuming your XML outside of
> the
> .NET platform. It is, overall, easier than working directly with the SQL
> "autogen" capabilities.
>
> --
> Gregory A. Beamer
> MVP; MCP: +I, SE, SD, DBA
> ***************************
> Think Outside the Box!
> ***************************
> "Mark Pustjens" wrote:
>
Thursday, March 8, 2012
Automated loading .CSVs into SQL Server
may be able to provide.
I have an incoming directory that recieves .csv files are regular
intervals. I need to be able to identify when these files arrive and
then load them into SQL Server into seperate tables overwriting the
current data. I was looking to use DTS, however i have never used it
before.
Could you assist or refer me to any useful sources?
Thank you in advance.Visit at www.sqldts.com
<dipeshpatel316@.gmail.com> wrote in message
news:1170760628.739101.96140@.a34g2000cwb.googlegroups.com...
>I am new to SQL Server and would be greatful for any assistance you
> may be able to provide.
> I have an incoming directory that recieves .csv files are regular
> intervals. I need to be able to identify when these files arrive and
> then load them into SQL Server into seperate tables overwriting the
> current data. I was looking to use DTS, however i have never used it
> before.
> Could you assist or refer me to any useful sources?
> Thank you in advance.
>
Automated loading .CSVs into SQL Server
may be able to provide.
I have an incoming directory that recieves .csv files are regular
intervals. I need to be able to identify when these files arrive and
then load them into SQL Server into seperate tables overwriting the
current data. I was looking to use DTS, however i have never used it
before.
Could you assist or refer me to any useful sources?
Thank you in advance.
Visit at www.sqldts.com
<dipeshpatel316@.gmail.com> wrote in message
news:1170760628.739101.96140@.a34g2000cwb.googlegro ups.com...
>I am new to SQL Server and would be greatful for any assistance you
> may be able to provide.
> I have an incoming directory that recieves .csv files are regular
> intervals. I need to be able to identify when these files arrive and
> then load them into SQL Server into seperate tables overwriting the
> current data. I was looking to use DTS, however i have never used it
> before.
> Could you assist or refer me to any useful sources?
> Thank you in advance.
>
Automated loading .CSVs into SQL Server
may be able to provide.
I have an incoming directory that recieves .csv files are regular
intervals. I need to be able to identify when these files arrive and
then load them into SQL Server into seperate tables overwriting the
current data. I was looking to use DTS, however i have never used it
before.
Could you assist or refer me to any useful sources?
Thank you in advance.Visit at www.sqldts.com
<dipeshpatel316@.gmail.com> wrote in message
news:1170760628.739101.96140@.a34g2000cwb.googlegroups.com...
>I am new to SQL Server and would be greatful for any assistance you
> may be able to provide.
> I have an incoming directory that recieves .csv files are regular
> intervals. I need to be able to identify when these files arrive and
> then load them into SQL Server into seperate tables overwriting the
> current data. I was looking to use DTS, however i have never used it
> before.
> Could you assist or refer me to any useful sources?
> Thank you in advance.
>
Automated Batch Job Over Numerous Data Files, How?
Hello, Everyone: Greetings!
I am new to Sql Server [Express version] and am not even sure I'm making the right choice. So here I am, seeking advice.
My database needs are nothing sophisticated. They just involve:
(a) create tens of thousands of separate data files each under a unique file name of up to 8 characters, with file names read in from a pre-determined file name list.
(b) store/insert VOLUMINOUS numerical data into each of the data files, with the data indexed by date&time, plus maybe one or two additional character or string fields.
(c) for each data file, retrieve a subset of its data, perform extensive numerical calculations, and then store the results in one or more separate corresponding files, e.g. if a file name in (b) is F12345, (c) creates F12345R1, F12345R2, F12345R3, etc. which stores different sets of calculated results.
Thus, this is purely a console application, doing a batch job, and requiring no graphical user interface. Both automation and speed are important here, due to the large number of data files that must be created and/or updated, and the very extensive numerical calculations on the data.
The goal is to automate the daily or weekly creation of each of the tens of thousands of Sql Server database files, insert fresh data (read in from a fresh ASCII file) into each file, numerically process the data and then store the results in one or more separate, corresponding result data files, with all the steps automated and without need for GUI. Once coding is done, the entire data processing session is expected to run for many hours, or even days, in an automated manner, and without human intervention.
What would be the most efficient way of doing this under Visual Basic Express (which is what I'm learning to use) by directly calling Sql Server Express without having to go through GUI to create database files? What is the proper interface utility or library to use to enable direct database function calls without the need to learn SQL language? Is Visual Basic and/or Sql Server even good choices for what I want to do? I want to be able to call the basic, simple database functions directly and simply from program code in a non-GUI, non-interactive manner for the tens of thousands of separate data files that will be used.
I really miss the good old days when one can do a straightforward batch job via a console application, with simple, direct calls to create new data files, insert and index fresh data, retrieve any subset of data to do extensive calculations, create new files to store the results, etc. all under automated program control and iterating through unlimited number of data files, until the job is finished, all without human intervention during processing.
Or am I missing something because all this can still be done simply and easily under VB and Sql Server? I've several books here about Visual Basic 2005 and Visual Basic 2005 Express, all showing how to create a database via a GUI utility. That's fine if one needs to create just one or two databases, but not hundreds, or even tens of thousands (as in my case) of them on the fly.
So, I am looking for the simplest and most direct database interface that will allow me to do the above under VB program code alone, and easily. For something as simple as I have described above, I don't think I should have to learn the SQL language or manually create each database file.
As you can see, I just want to get some heavy duty numerical processing job done over tens of thousands of data files as simply and efficiently as possible, and with as little fanciful detour as possible. So, ironically, I am trying to use Visual Basic without being cluttered by having to learn its "Visual" aspects, yet GUI is what most VB books devote to or emphasize heavily. Similarly, I would much rather use simple, "lean and mean", direct database function calls than having to learn a new vocabulary of "English-like" SQL language.
Yes, I'm not used to this tedious detour of learning the GUI aspect of VB, or learning the Structured Query Language of Sql Server, just to try to do something simple that I need to do in batch mode via a console application.
Are there any good books or other helpful URLs that will help a guy like me? Am I even using the wrong language and the wrong database to do what I want to do? What are the better alternatives, if any? Any advice, experience and pointers on any of the above issues raised would be very much appreciated. Thank you!
Regards,
ConsoleApp
You've asked a fairly large, fairly non-trivial question...
SQL Server 2005 Express Edition is a good place to start, as you've surmised. Depending on your background and expertise, Visual C# or Visual Basic from Visual Studio 2005 would be a great programming environment. You'll be able to create a new database and add tables, as well as use the database project for managing the TSQL scripts that can be used to create these database objects on the fly when your application runs. Depending on your indexing vs. partitioning decision, you may only ever need to create the database and tables one time.
In both C# and Visual Basic, a console app that does what you're asking it to do should be possible. My only concern (follow up in one of the Visual Studio forums for an autoritative answer and guide to this) is the ability to use some of the great data designer tools in a UI-less application -- the designers and associated runtime controls are targeted toward a rich client application and thus assume a windowed application. For that reason, you may find that prototyping your application as a Windows Forms application may be easier to start with and then transition to a non-UI application at a later date (note that a windowed app can still serve as the container for a batch processing type app such as this...what you're asking for is an application that doesn't display UI that blocks the progress of the job).
As to some of the other aspects of your application, you will more likely want to create lots of tables, not databases. There is a very rough correlation between databases and directories and between tables and files...that general mindset will probably help you transition your mental model. Also, look at the justification behind partitioning your data into many, many files...in a database, you can use indexing to differentiate these groupings of data (e.g. SELECT * FROM [Data] WHERE [SensorID]=4). For extremely large data sets, there may still be some value in partitioning, but I suspect that this is a tertiary consideration at this point.
And finally, your processed data could be stored in additional tables much as your input data was. On these tables you could then do additional analysis/reporting...
Are you parsing out non-contiguous subsets of data (e.g. [1-10], [11-20], [21-30] or is it [1,2,4,10,12], [2, 20,21,30,33], etc) on a regular basis? Or do you process data files in their entirety?
Automated Batch Job Over Numerous Data Files, How?
Hello, Everyone: Greetings!
I am new to Sql Server [Express version] and am not even sure I'm making the right choice. So here I am, seeking advice.
My database needs are nothing sophisticated. They just involve:
(a) create tens of thousands of separate data files each under a unique file name of up to 8 characters, with file names read in from a pre-determined file name list.
(b) store/insert VOLUMINOUS numerical data into each of the data files, with the data indexed by date&time, plus maybe one or two additional character or string fields.
(c) for each data file, retrieve a subset of its data, perform extensive numerical calculations, and then store the results in one or more separate corresponding files, e.g. if a file name in (b) is F12345, (c) creates F12345R1, F12345R2, F12345R3, etc. which stores different sets of calculated results.
Thus, this is purely a console application, doing a batch job, and requiring no graphical user interface. Both automation and speed are important here, due to the large number of data files that must be created and/or updated, and the very extensive numerical calculations on the data.
The goal is to automate the daily or weekly creation of each of the tens of thousands of Sql Server database files, insert fresh data (read in from a fresh ASCII file) into each file, numerically process the data and then store the results in one or more separate, corresponding result data files, with all the steps automated and without need for GUI. Once coding is done, the entire data processing session is expected to run for many hours, or even days, in an automated manner, and without human intervention.
What would be the most efficient way of doing this under Visual Basic Express (which is what I'm learning to use) by directly calling Sql Server Express without having to go through GUI to create database files? What is the proper interface utility or library to use to enable direct database function calls without the need to learn SQL language? Is Visual Basic and/or Sql Server even good choices for what I want to do? I want to be able to call the basic, simple database functions directly and simply from program code in a non-GUI, non-interactive manner for the tens of thousands of separate data files that will be used.
I really miss the good old days when one can do a straightforward batch job via a console application, with simple, direct calls to create new data files, insert and index fresh data, retrieve any subset of data to do extensive calculations, create new files to store the results, etc. all under automated program control and iterating through unlimited number of data files, until the job is finished, all without human intervention during processing.
Or am I missing something because all this can still be done simply and easily under VB and Sql Server? I've several books here about Visual Basic 2005 and Visual Basic 2005 Express, all showing how to create a database via a GUI utility. That's fine if one needs to create just one or two databases, but not hundreds, or even tens of thousands (as in my case) of them on the fly.
So, I am looking for the simplest and most direct database interface that will allow me to do the above under VB program code alone, and easily. For something as simple as I have described above, I don't think I should have to learn the SQL language or manually create each database file.
As you can see, I just want to get some heavy duty numerical processing job done over tens of thousands of data files as simply and efficiently as possible, and with as little fanciful detour as possible. So, ironically, I am trying to use Visual Basic without being cluttered by having to learn its "Visual" aspects, yet GUI is what most VB books devote to or emphasize heavily. Similarly, I would much rather use simple, "lean and mean", direct database function calls than having to learn a new vocabulary of "English-like" SQL language.
Yes, I'm not used to this tedious detour of learning the GUI aspect of VB, or learning the Structured Query Language of Sql Server, just to try to do something simple that I need to do in batch mode via a console application.
Are there any good books or other helpful URLs that will help a guy like me? Am I even using the wrong language and the wrong database to do what I want to do? What are the better alternatives, if any? Any advice, experience and pointers on any of the above issues raised would be very much appreciated. Thank you!
Regards,
ConsoleApp
You've asked a fairly large, fairly non-trivial question...
SQL Server 2005 Express Edition is a good place to start, as you've surmised. Depending on your background and expertise, Visual C# or Visual Basic from Visual Studio 2005 would be a great programming environment. You'll be able to create a new database and add tables, as well as use the database project for managing the TSQL scripts that can be used to create these database objects on the fly when your application runs. Depending on your indexing vs. partitioning decision, you may only ever need to create the database and tables one time.
In both C# and Visual Basic, a console app that does what you're asking it to do should be possible. My only concern (follow up in one of the Visual Studio forums for an autoritative answer and guide to this) is the ability to use some of the great data designer tools in a UI-less application -- the designers and associated runtime controls are targeted toward a rich client application and thus assume a windowed application. For that reason, you may find that prototyping your application as a Windows Forms application may be easier to start with and then transition to a non-UI application at a later date (note that a windowed app can still serve as the container for a batch processing type app such as this...what you're asking for is an application that doesn't display UI that blocks the progress of the job).
As to some of the other aspects of your application, you will more likely want to create lots of tables, not databases. There is a very rough correlation between databases and directories and between tables and files...that general mindset will probably help you transition your mental model. Also, look at the justification behind partitioning your data into many, many files...in a database, you can use indexing to differentiate these groupings of data (e.g. SELECT * FROM [Data] WHERE [SensorID]=4). For extremely large data sets, there may still be some value in partitioning, but I suspect that this is a tertiary consideration at this point.
And finally, your processed data could be stored in additional tables much as your input data was. On these tables you could then do additional analysis/reporting...
Are you parsing out non-contiguous subsets of data (e.g. [1-10], [11-20], [21-30] or is it [1,2,4,10,12], [2, 20,21,30,33], etc) on a regular basis? Or do you process data files in their entirety?
Saturday, February 25, 2012
Autogrowth for MDF and LDF files
Thanks,
CarlosI have a SQL 2005 DB that its MDF file is growing at a rate of 1 GB per day, I currently have it set up to unrestricted growth by 500 MB. Should I increase that growth to 1 GB? what would the impact of this change be? what are best practices when it comes to setting up autogrowth for MDF and LDF files?
Thanks,
Carlos
I generally allow dbs that are smaller that 20 GB to autogrow by the default setting (10%). But once they get above that size, I manage them manually and ensure that there is enough empty space in the datafile to get through until the next maintenance window.
Growing a data file in SQL 2005 is not as expensive (IO wise) as it was in SQL 2000, but I still think you want to keep a closer eye on things once they get above 20 GB. 20 GB is admittedly arbitrary. If you have space issues, you might consider a lower threshold.
Regards,
hmscott|||It depends on what your database is used for. Mine are configured to grow by several hundred MB\ a few GB but that is because there are small numbers of massive modifications. An OLTP database should not, IMHO, be growing that much each time. The user that submitted the modification that triggers a 500MB growth could be twiddling their thumbs for quite some time cursing the system as they do.
I too would manage the growth at peak periods with a view to eliminating\ reducing autogrowth as much as possible.|||Your file growth is because of your 500MB setting. When SQL starts to run out of space it will adjust by 500MB. If has to adjust twice a day, there's your 1GB. I think you're fine where you're at. Just make sure you have enough drive space. I'm sure your growth will plateau.
Sunday, February 19, 2012
Auto UpLoad
My DD Subscription produces .PDF files of the reports , I would like to
upload them automatically so the user can see the files via Report Manager.
Has anyone had to do thi yet?
Many thanks in advance
PaulIf I understand you correctly, (and I'm assuming some things), you have a
report(s) that take some time to generate and you want to generate them in
the background but enable the users to see them through the web-page without
the latency?
If this is correct, set up the history so that it creates the report at a
regular interval, then when someone requests the report, it will pull it from
the history.
"PaulQld" wrote:
> Is there a way to automate the upload of reports (.PDF's).
> My DD Subscription produces .PDF files of the reports , I would like to
> upload them automatically so the user can see the files via Report Manager.
> Has anyone had to do thi yet?
> Many thanks in advance
> Paul
Thursday, February 16, 2012
Auto scripting SPs, perhaps concatenating text fields...
Hi,
I'm trying to get the text of all my SPs saved into text (*.sql) files. My first thought was to use sp_helptext and bcp the table to a text file but this isn't working (see my other post) so thought I'd try another method.
I can get the code from syscomment.text and concatenate the varchar(8000) field together in a text field. Unfortunately this isn't as easy as just text = text + newtext, how is this done?
Or am I doing it all comletely the wrong way? BTW, I have over 150 SPs so I can't save them individually.
Thanks!
Nick
Why don't you use EM or SSMS for that? Both utilities have feature to scrip database objects.|||I've just found the "create one file per object" option. Thanks gavrilenko_s!
Nick
Sunday, February 12, 2012
Auto importing files
Hi, I am running sqlserver and I have a directory taking file dumps, what I need to do is have the database auto import the file into a table, but I want to avoid using asp.net (I had considered using a filecatcher), I want to limit what I use to VB or sqlserver components so other people in my area can also support it.
any help or pointers in the right direction would be appreciated.
which version of sql server you have? if its sql server 2000 you can try DTS and if its 2005 try 2005. BTB what type of file you receive?
Madhu
|||I am using 2005|||Hope its not Express. if its express SSIS will not work. otherwise , you can create SSIS for file porting and you can schedule as u want using sql server job
Madhu
Friday, February 10, 2012
Auto grow
AUTO GROW since the drive it was pointing to does not have enough space
available. The secondary file was set to AUTO GROW and the drive space has
enough available space. Provided that the secondary file has plenty of room
for new insert will there be any case that the new INSERT will fail? I am
trying to troubleshoot an issue on which the INSERT was failing but when I
was reset the primary file to AUTO GROW the problem disappeared.
Yes SQL server will write proportionally to the files according to their
allocated sizes. You will have to move the file that can no longer grow to
a drive where it has space to grow. Or you may put the content of the file
that can no longer grow into the other file(s).
hth
Quentin
"J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
>I have two files set up for one filegroup and the primary file was set to
>NO
> AUTO GROW since the drive it was pointing to does not have enough space
> available. The secondary file was set to AUTO GROW and the drive space
> has
> enough available space. Provided that the secondary file has plenty of
> room
> for new insert will there be any case that the new INSERT will fail? I am
> trying to troubleshoot an issue on which the INSERT was failing but when I
> was reset the primary file to AUTO GROW the problem disappeared.
|||If there are two files in a database, and one file is full without the AUTO
GROW setting, then SQL Server should only try to allocate from the other
file. So most likely the insert will not fail as long as the other file can
auto grow or has free space.
From a performance point of view, this is not the ideal situation for your
database. We recommend that the database files are configured so that each
file has some free space in it.
--
Wei Xiao [MSFT]
SQL Server Storage Engine Development
http://blogs.msdn.com/weix
"Quentin Ran" <remove_this_qran2@.yahoo.com> wrote in message
news:#um$101dFHA.2960@.TK2MSFTNGP10.phx.gbl...
> Yes SQL server will write proportionally to the files according to their
> allocated sizes. You will have to move the file that can no longer grow
to
> a drive where it has space to grow. Or you may put the content of the
file[vbcol=seagreen]
> that can no longer grow into the other file(s).
> hth
> Quentin
> "J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
> news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
am[vbcol=seagreen]
I
>
Auto grow
AUTO GROW since the drive it was pointing to does not have enough space
available. The secondary file was set to AUTO GROW and the drive space has
enough available space. Provided that the secondary file has plenty of room
for new insert will there be any case that the new INSERT will fail? I am
trying to troubleshoot an issue on which the INSERT was failing but when I
was reset the primary file to AUTO GROW the problem disappeared.Yes SQL server will write proportionally to the files according to their
allocated sizes. You will have to move the file that can no longer grow to
a drive where it has space to grow. Or you may put the content of the file
that can no longer grow into the other file(s).
hth
Quentin
"J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
>I have two files set up for one filegroup and the primary file was set to
>NO
> AUTO GROW since the drive it was pointing to does not have enough space
> available. The secondary file was set to AUTO GROW and the drive space
> has
> enough available space. Provided that the secondary file has plenty of
> room
> for new insert will there be any case that the new INSERT will fail? I am
> trying to troubleshoot an issue on which the INSERT was failing but when I
> was reset the primary file to AUTO GROW the problem disappeared.|||If there are two files in a database, and one file is full without the AUTO
GROW setting, then SQL Server should only try to allocate from the other
file. So most likely the insert will not fail as long as the other file can
auto grow or has free space.
From a performance point of view, this is not the ideal situation for your
database. We recommend that the database files are configured so that each
file has some free space in it.
--
Wei Xiao [MSFT]
SQL Server Storage Engine Development
http://blogs.msdn.com/weix
"Quentin Ran" <remove_this_qran2@.yahoo.com> wrote in message
news:#um$101dFHA.2960@.TK2MSFTNGP10.phx.gbl...
> Yes SQL server will write proportionally to the files according to their
> allocated sizes. You will have to move the file that can no longer grow
to
> a drive where it has space to grow. Or you may put the content of the
file
> that can no longer grow into the other file(s).
> hth
> Quentin
> "J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
> news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
am[vbcol=seagreen]
I[vbcol=seagreen]
>
Auto grow
AUTO GROW since the drive it was pointing to does not have enough space
available. The secondary file was set to AUTO GROW and the drive space has
enough available space. Provided that the secondary file has plenty of room
for new insert will there be any case that the new INSERT will fail? I am
trying to troubleshoot an issue on which the INSERT was failing but when I
was reset the primary file to AUTO GROW the problem disappeared.Yes SQL server will write proportionally to the files according to their
allocated sizes. You will have to move the file that can no longer grow to
a drive where it has space to grow. Or you may put the content of the file
that can no longer grow into the other file(s).
hth
Quentin
"J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
>I have two files set up for one filegroup and the primary file was set to
>NO
> AUTO GROW since the drive it was pointing to does not have enough space
> available. The secondary file was set to AUTO GROW and the drive space
> has
> enough available space. Provided that the secondary file has plenty of
> room
> for new insert will there be any case that the new INSERT will fail? I am
> trying to troubleshoot an issue on which the INSERT was failing but when I
> was reset the primary file to AUTO GROW the problem disappeared.|||If there are two files in a database, and one file is full without the AUTO
GROW setting, then SQL Server should only try to allocate from the other
file. So most likely the insert will not fail as long as the other file can
auto grow or has free space.
From a performance point of view, this is not the ideal situation for your
database. We recommend that the database files are configured so that each
file has some free space in it.
--
Wei Xiao [MSFT]
SQL Server Storage Engine Development
http://blogs.msdn.com/weix
"Quentin Ran" <remove_this_qran2@.yahoo.com> wrote in message
news:#um$101dFHA.2960@.TK2MSFTNGP10.phx.gbl...
> Yes SQL server will write proportionally to the files according to their
> allocated sizes. You will have to move the file that can no longer grow
to
> a drive where it has space to grow. Or you may put the content of the
file
> that can no longer grow into the other file(s).
> hth
> Quentin
> "J Pacquiao" <JPacquiao@.discussions.microsoft.com> wrote in message
> news:5F4B85B2-1F6D-449C-8846-1AFE09F9168F@.microsoft.com...
> >I have two files set up for one filegroup and the primary file was set to
> >NO
> > AUTO GROW since the drive it was pointing to does not have enough space
> > available. The secondary file was set to AUTO GROW and the drive space
> > has
> > enough available space. Provided that the secondary file has plenty of
> > room
> > for new insert will there be any case that the new INSERT will fail? I
am
> > trying to troubleshoot an issue on which the INSERT was failing but when
I
> > was reset the primary file to AUTO GROW the problem disappeared.
>