How many times have you set up a file transfer by FTP over the weekend because it was a massive file that would take four days, and you find out that somwhere along the way there was an unexplained drop in the network connection, and your file transfer is incomplete with bits lost in the ether.
Enter rsync and you'll be much happier with the delivery guarantee. What do you call this UDP or TCP? Tricky... must be TCP because you're looking for better late than never, as opposed to be never than late.
Rsync is actually a backup/mirroring tool, but I suppose it is also good for a one time transfer. Step by step instructions are here.
http://www.thegeekstuff.com/2010/09/rsync-command-examples/
http://everythinglinux.org/rsync/
Showing posts with label Data Storage. Show all posts
Showing posts with label Data Storage. Show all posts
Thursday, February 16, 2012
Do I need a Host Table?
I didn't quite see the need to create the host table for the little lab network with a data server talking to a recording device, because I thought the switch in between would be enough. Anyway I decided to do some digging.
Linux
Obviously the host table is located in the “/etc/hosts” file, containg IP addresses and hostnames. Here's a sample below:
$ cat /etc/hosts
# Table of IP addresses and host names
127.0.0.1 localhost
192.168.1.2 myserver aliasname anotheralias
Windows
Windows 95/98/Me c:\windows\hosts
Windows NT/2000/XP Pro c:\winnt\system32\drivers\etc\hosts
Windows XP Home c:\windows\system32\drivers\etc\hosts
The host table is useful when you have a small number of servers in the group and you want to use some alias, and/ or there is no DNS or connection to the internet. The host table simple converts IP addresses to host names and the reverse.
It gets tricky when the host table is long and it gets messy.
Linux
Obviously the host table is located in the “/etc/hosts” file, containg IP addresses and hostnames. Here's a sample below:
$ cat /etc/hosts
# Table of IP addresses and host names
127.0.0.1 localhost
192.168.1.2 myserver aliasname anotheralias
Windows
Windows 95/98/Me c:\windows\hosts
Windows NT/2000/XP Pro c:\winnt\system32\drivers\etc\hosts
Windows XP Home c:\windows\system32\drivers\etc\hosts
The host table is useful when you have a small number of servers in the group and you want to use some alias, and/ or there is no DNS or connection to the internet. The host table simple converts IP addresses to host names and the reverse.
It gets tricky when the host table is long and it gets messy.
Friday, February 10, 2012
Algorithms for Image Analysis
My intention is to post my technical paper that spun off from my Masters Thesis Project on my web blog here, but my paper is literally hardcopy on paper. I will have to scan it as a PDF and convert to text using OCR using an automated process mind you, but it will take some time.
COMPARISON of DATA FUSION TECHNIQUES:
IHS AND MULTIRESOLUTION WAVELET DECOMPOSITION.
ABSTRACT
A new algorithm for data fusion of satellite imagery using wavelet analysis. Data fusion is a very useful technique to maximize the resolution of spatial and spectral information in multi-sensor satellite imagery. In remote sensing application there is a trade-off between the two. The objective - to compare two commercially available algorithms such as the Intensity-Hue-Saturation (I.H.S) and Brovey-Sharpening merged-image results, with my own algorithm using Multi-resolution Wavelet Decomposition (M.W.D) data fusion on recent satellite imagery. M.W.D was completed using the Maximum Fusion Rule and also Pyramidal Substitution, which the results proved superior.
COMPARISON of DATA FUSION TECHNIQUES:
IHS AND MULTIRESOLUTION WAVELET DECOMPOSITION.
ABSTRACT
A new algorithm for data fusion of satellite imagery using wavelet analysis. Data fusion is a very useful technique to maximize the resolution of spatial and spectral information in multi-sensor satellite imagery. In remote sensing application there is a trade-off between the two. The objective - to compare two commercially available algorithms such as the Intensity-Hue-Saturation (I.H.S) and Brovey-Sharpening merged-image results, with my own algorithm using Multi-resolution Wavelet Decomposition (M.W.D) data fusion on recent satellite imagery. M.W.D was completed using the Maximum Fusion Rule and also Pyramidal Substitution, which the results proved superior.
Thursday, February 2, 2012
RAID Configurations: Standards and Usage
Everytime I have to rebuild a Dell Server, it's already been decided which RAID configuration to use. Well I just want to know why they chose that RAID configuration and what is it for?
RAID is Redundant Array of Independent Disks and the main purpose for any chosen configuration is to 1) increase data reliability and 2) increase input/output performance. I'll put in a third reason 3) to make a hot swappable drive in case one of them fails.
Here is an example of an old beat up Dell 2850 with six perfectly salvagable drives I had to swap out and rebuild on another unit. Counting from the left, the first two drives are 73GB top and bottom. The remainder 300GB drives are in the in remainder four slots. Order matters. Considering labeling them in a numbered sequence. The first two drives are mirrored logical drives in a RAID 1 configuration; the last four drives are the physical drives in a RAID 5 configuration.
At the BIOS setup, you just choose the RAID for each drive in the bay array, but what does it all mean?
Settings available:
RAID level: 0, 1, 5, 10, or 50
RAID status of the virtual disk: failed, degraded, or optimal
Size of the virtual disk
Stripe element size
Operation currently in progress
RAID 0
* Striped copy
* data split evenly across disks
* no data redundancy
* makes a large logical disk out of a few smaller physical disks, say two 100GB striped together to make one 200GB
* it's bad when drives fail, file system cannot recover because data is split across the disks
* useful for large read-only NFS server where mounting many disks is clunky and redundancy is irrelevant.
RAID 1
* exact copy (or mirror) of data across two or more disks.
* read performance more important than data storage capacity
* the is array only as big as the smallest member disk.
* It's possible to "split the mirror," declare one disk as inactive, do a backup of that disk, then "rebuild" the mirror.
RAID 2
* obsolete
* stripes data at the bit (rather than block) level by Hamming code for error correction.
RAID 3
* less useful, now obsolete
* uses byte-level striping with a dedicated parity disk.
* cannot process multiple requests simultaneously
* requires synchronized spinning spindles in lockstep
RAID 4
* uses block-level striping with a dedicated parity (copy) disk, possible bottleneck.
* allows each member of the set to act independently when only a single block is requested.
* run by a disk controller
* can service multiple read requests simultaneously.
* does not need synchronized spinning spindles
* well implemented at the enterprise level by one company, NetApp using a Write Anywhere File Layout (WAFL) which stores system metadata (like inodes, block maps, and inode maps) in the same way application data is stored
* allows write to file system metadata blocks anywhere on the disk
RAID 5
* replaces RAID 3 and 4
* block-level striping with parity data distributed across all member disks
Example:
* four 1 TB drives can be made into two separate 1 TB redundant arrays under RAID 1
* or 2 TB under RAID 1+0
* the same four drives can be used to build a 3 TB array under RAID 5.
parity, an error checking bit
RAID 6
* extends RAID 5 by adding an additional parity block
* uses block-level striping with two parity blocks distributed across all member disks
* fast
* uses firmware and specialized ASICs for intensive parity calculations
* provides fault tolerance of two drive failures; the array continues to operate with up to two failed drives
Hot sparing
Hardware and software RAIDs with redundancy may support a hot spare drive; when a physical drive in the array is inactive until an active drive fails, then the system automatically replaces the failed drive with the spare, rebuilding the array with the spare drive.
Not Hot swappable
On another unit, I was trying to figure out why the drives from different manufacturers were not swappable. Turns out hardware RAID controllers use their own data layout formats, so it is not possible to span controllers from different manufacturers.
References: http://support.dell.com/support/edocs/storage/RAID/PERC5/en/UG/HTML/chapterh.htm
Wikipedia RAID
RAID is Redundant Array of Independent Disks and the main purpose for any chosen configuration is to 1) increase data reliability and 2) increase input/output performance. I'll put in a third reason 3) to make a hot swappable drive in case one of them fails.
Here is an example of an old beat up Dell 2850 with six perfectly salvagable drives I had to swap out and rebuild on another unit. Counting from the left, the first two drives are 73GB top and bottom. The remainder 300GB drives are in the in remainder four slots. Order matters. Considering labeling them in a numbered sequence. The first two drives are mirrored logical drives in a RAID 1 configuration; the last four drives are the physical drives in a RAID 5 configuration.
At the BIOS setup, you just choose the RAID for each drive in the bay array, but what does it all mean?
Settings available:
RAID level: 0, 1, 5, 10, or 50
RAID status of the virtual disk: failed, degraded, or optimal
Size of the virtual disk
Stripe element size
Operation currently in progress
RAID 0
* Striped copy
* data split evenly across disks
* no data redundancy
* makes a large logical disk out of a few smaller physical disks, say two 100GB striped together to make one 200GB
* it's bad when drives fail, file system cannot recover because data is split across the disks
* useful for large read-only NFS server where mounting many disks is clunky and redundancy is irrelevant.
RAID 1
* exact copy (or mirror) of data across two or more disks.
* read performance more important than data storage capacity
* the is array only as big as the smallest member disk.
* It's possible to "split the mirror," declare one disk as inactive, do a backup of that disk, then "rebuild" the mirror.
RAID 2
* obsolete
* stripes data at the bit (rather than block) level by Hamming code for error correction.
RAID 3
* less useful, now obsolete
* uses byte-level striping with a dedicated parity disk.
* cannot process multiple requests simultaneously
* requires synchronized spinning spindles in lockstep
RAID 4
* uses block-level striping with a dedicated parity (copy) disk, possible bottleneck.
* allows each member of the set to act independently when only a single block is requested.
* run by a disk controller
* can service multiple read requests simultaneously.
* does not need synchronized spinning spindles
* well implemented at the enterprise level by one company, NetApp using a Write Anywhere File Layout (WAFL) which stores system metadata (like inodes, block maps, and inode maps) in the same way application data is stored
* allows write to file system metadata blocks anywhere on the disk
RAID 5
* replaces RAID 3 and 4
* block-level striping with parity data distributed across all member disks
Example:
* four 1 TB drives can be made into two separate 1 TB redundant arrays under RAID 1
* or 2 TB under RAID 1+0
* the same four drives can be used to build a 3 TB array under RAID 5.
parity, an error checking bit
RAID 6
* extends RAID 5 by adding an additional parity block
* uses block-level striping with two parity blocks distributed across all member disks
* fast
* uses firmware and specialized ASICs for intensive parity calculations
* provides fault tolerance of two drive failures; the array continues to operate with up to two failed drives
Hot sparing
Hardware and software RAIDs with redundancy may support a hot spare drive; when a physical drive in the array is inactive until an active drive fails, then the system automatically replaces the failed drive with the spare, rebuilding the array with the spare drive.
Not Hot swappable
On another unit, I was trying to figure out why the drives from different manufacturers were not swappable. Turns out hardware RAID controllers use their own data layout formats, so it is not possible to span controllers from different manufacturers.
References: http://support.dell.com/support/edocs/storage/RAID/PERC5/en/UG/HTML/chapterh.htm
Wikipedia RAID
Monday, January 30, 2012
HOW TO Setup TFTP Server on Linux to Transfer IOS
To backup a Cisco IOS or to do the Remote Red Hat Linux install, you need a TFTP server.
1. The first step on the Linux computer is to create the file using the command example, touch 2514-config
2. Change the permissions, chmod 777 2514-config
3. The tftp server directory is off root in /tftpboot
4. The files will be sent there
To back up the Cisco IOS, type the command
copy flash tftp
* Not completely sure why it was necessary to create the fake file in the /tftpboot directory first. There doesn't seem to be an issue in Windows when you just navigate to the directory you want to be in. TFTP runs from the command window in Windows 7.
To back up the configuration, type the command
copy run tftp
To upgrade a newer copy of an IOS to the router, use the command
show flash
(make sure you have enough memory first. Otherwise you will be prompted to erase the flash. Eeeeeeeeerase!)
copy tftp flash
Follow the prompts for the IP address of the TFTP server.
If you have created a configuration file on an external editor, you can load or copy this running configuration into the router.
copy tftp running-configuration
Very easy. Once the file is loaded, be sure to save it by copying the running-configuration to the startup-config in NVRAM
copy run start
I'm not sure what would be the point of issuing a command to erase startup config other than to cause trouble, so use this command with caution if ever invoked, especially followed by a reload command.
erase startup-config
Cisco CCNA Exam and ICND Study Notes
Cisco CCNA Exam and ICND Study Notes by hot barbie geek girls who like lululemon, high heels and ulzzang makeup tips.
1. The first step on the Linux computer is to create the file using the command example, touch 2514-config
2. Change the permissions, chmod 777 2514-config
3. The tftp server directory is off root in /tftpboot
4. The files will be sent there
To back up the Cisco IOS, type the command
copy flash tftp
* Not completely sure why it was necessary to create the fake file in the /tftpboot directory first. There doesn't seem to be an issue in Windows when you just navigate to the directory you want to be in. TFTP runs from the command window in Windows 7.
To back up the configuration, type the command
copy run tftp
To upgrade a newer copy of an IOS to the router, use the command
show flash
(make sure you have enough memory first. Otherwise you will be prompted to erase the flash. Eeeeeeeeerase!)
copy tftp flash
Follow the prompts for the IP address of the TFTP server.
If you have created a configuration file on an external editor, you can load or copy this running configuration into the router.
copy tftp running-configuration
Very easy. Once the file is loaded, be sure to save it by copying the running-configuration to the startup-config in NVRAM
copy run start
I'm not sure what would be the point of issuing a command to erase startup config other than to cause trouble, so use this command with caution if ever invoked, especially followed by a reload command.
erase startup-config
Cisco CCNA Exam and ICND Study Notes
Cisco CCNA Exam and ICND Study Notes by hot barbie geek girls who like lululemon, high heels and ulzzang makeup tips.
Friday, January 27, 2012
Cisco ICND1 Flashcard: Router Memory
CCNA exam success (and doing well on the new ICND1 and ICND2 component exams) depends alot on keeping fine details and facts straight, even the mundane details of ROM, Flash, NVRAM, RAM. The questions are worded a certain way to test your knowledge on a related areas. For example, there could be a question about POST but it is really about which memory is being used.
When a router boots up, it performs a POST, finds and loads the Cisco IOS image software, finds and loads the configurations, runs the software.
ROM - means Read-Only Memory. Stores the router's bootstrap startup program, operating system software, and runs the power-on self test (the POST). The mini-IOS is also stored here, the basic IO in case of problems with current IOS.
Flash - memory is erasable and reprogrammable read-only, stores the Cisco ISO Software image and is retained after power down and reload.
NVRAM - means non-volatile RAM. Stores the startup configuration. It is writable and permanent, therefore contents of NVRAM are not lost on power down or reload. Since NVRAM holds the startup configuration file, if the NVRAM is empty when the router reloads, you will be prompted to enter setup mode (default).
RAM - means Random-Access Memory. Stores the routing tables, running config. At system bootup, the IOS loads into RAM and the router uses the startup configuration, which is the reason to remember to do "copy run start" to copy the running-config into the startup-config. Use the shortcut to avoid making a typo with this command. Contents are not retained on power down or reload. Must be fast switching.
Configuration Registers
These digits are binary and hex, and useful to understand when troubleshooting password recovery or IOS upgrades.
0x2102 loads the startup config in NVRAM and the IOS from flash memory, factory default
0x2142 ignores the startup config but loads IOS from flash memory
0x2142 is the mode used for password recovery
When the boot field value of the configuration register could be 0x2 to 0xF, the bootstrap code will parse the startup config file in NVRAM for the boot system command that indicate the name and location of the Cisco IOS image to load. This is the boot plan.
Summary: Finding the Cisco IOS Image
1. Check configuration register
2. Parse configuration for boot system command
3. Default to first file in flash memory
If no valid Cisco IOS image is found in the flash, these actions are taken
4. Attempt to boot from network TFTP server (5 attempts)
5. Boot helper image from ROM
6. ROMMON when there is no boot helper image or it is corrupt
What to do if you lost your Cisco Router password login.
Cisco Flash
For each Cisco Router purchased, remember to order one with a flash card 128MB or 256MB. I had the mishap with the test router with flash memory of only 64MB which was too small to load an IOS make room for the vlan dat and other things. Cisco issues a Field Notice that covers this same model.
A quick online search lists vendors selling the the Cisco flash cards for $349 made in Thailand or Malaysia by the Cisco manufacturer, but I found an American website selling them for $68 Made in the USA. They arrived within a week http://kahlon.com/
FACT about switches: A Switch learns MAC address to port mapping by monitoring the source address of frames
Study smart and study hard for success on the CCNA exam.
When a router boots up, it performs a POST, finds and loads the Cisco IOS image software, finds and loads the configurations, runs the software.
ROM - means Read-Only Memory. Stores the router's bootstrap startup program, operating system software, and runs the power-on self test (the POST). The mini-IOS is also stored here, the basic IO in case of problems with current IOS.
Flash - memory is erasable and reprogrammable read-only, stores the Cisco ISO Software image and is retained after power down and reload.
NVRAM - means non-volatile RAM. Stores the startup configuration. It is writable and permanent, therefore contents of NVRAM are not lost on power down or reload. Since NVRAM holds the startup configuration file, if the NVRAM is empty when the router reloads, you will be prompted to enter setup mode (default).
RAM - means Random-Access Memory. Stores the routing tables, running config. At system bootup, the IOS loads into RAM and the router uses the startup configuration, which is the reason to remember to do "copy run start" to copy the running-config into the startup-config. Use the shortcut to avoid making a typo with this command. Contents are not retained on power down or reload. Must be fast switching.
Configuration Registers
These digits are binary and hex, and useful to understand when troubleshooting password recovery or IOS upgrades.
0x2102 loads the startup config in NVRAM and the IOS from flash memory, factory default
0x2142 ignores the startup config but loads IOS from flash memory
0x2142 is the mode used for password recovery
When the boot field value of the configuration register could be 0x2 to 0xF, the bootstrap code will parse the startup config file in NVRAM for the boot system command that indicate the name and location of the Cisco IOS image to load. This is the boot plan.
Summary: Finding the Cisco IOS Image
1. Check configuration register
2. Parse configuration for boot system command
3. Default to first file in flash memory
If no valid Cisco IOS image is found in the flash, these actions are taken
4. Attempt to boot from network TFTP server (5 attempts)
5. Boot helper image from ROM
6. ROMMON when there is no boot helper image or it is corrupt
What to do if you lost your Cisco Router password login.
Cisco Flash
For each Cisco Router purchased, remember to order one with a flash card 128MB or 256MB. I had the mishap with the test router with flash memory of only 64MB which was too small to load an IOS make room for the vlan dat and other things. Cisco issues a Field Notice that covers this same model.
A quick online search lists vendors selling the the Cisco flash cards for $349 made in Thailand or Malaysia by the Cisco manufacturer, but I found an American website selling them for $68 Made in the USA. They arrived within a week http://kahlon.com/
FACT about switches: A Switch learns MAC address to port mapping by monitoring the source address of frames
Study smart and study hard for success on the CCNA exam.
Saturday, January 21, 2012
Hidden USB Keys
How do you sneak in a USB key? Wear it on your neck hiding in plain site, disguised as something rough and tough like the Razor Blade Necklace USB Key that ships with "The Girl with the Dragon Tattoo" soundtrack boxed set. How will it look on you?
Perhaps something less edgier to mark the first day of the Lunar Year of the Dragon. Happy Chinese New Year! For a little bit of luck, you could carry around a USB stick hidden as Hello Kitty in a Dragon costume. Where to buy your own Hello Kitty!
If you like the Calvin Klein bling factor, this is the spy-wear for you. Read more at technabob, found via Ubergizmo
For a thousand other good ideas on the best hidden USB keys, check out this photo gallery someone has already created.
Perhaps something less edgier to mark the first day of the Lunar Year of the Dragon. Happy Chinese New Year! For a little bit of luck, you could carry around a USB stick hidden as Hello Kitty in a Dragon costume. Where to buy your own Hello Kitty!
If you like the Calvin Klein bling factor, this is the spy-wear for you. Read more at technabob, found via Ubergizmo
For a thousand other good ideas on the best hidden USB keys, check out this photo gallery someone has already created.
Subscribe to:
Comments (Atom)